Guru Sattanathan, Confluent, Senior Solutions Engineer Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats! Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts? If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today. In this session, we will go in this order: -Recap on Enterprise Integration technologies -What are the key flaws & What needs improvement? -What is Apache Kafka? -Rethinking Integration using Apache Kafka https://www.meetup.com/KafkaMelbourne/events/280590162/

1. @avoguru
Death of the dumb pipes
Its time to move towards Apache Kafka® for integration problems
Gnanaguru (Guru) Sattanathan
Senior Solutions Engineer

2. Focus for today
2
Recap
How Middleware
technologies helped
build integrated
applications ?
Key flaws
Why traditional
middleware
technologies are not
suitable anymore ?
Apache Kafka
What is Kafka ?
Why it is so different &
disruptive ?
Future
How organisations are
building integrations
today ? Where are we
heading ?

3. Recap - Middleware
I call it - Traditional Integration Products
3

4. @avoguru
Need for Integration
4
Disconnected applications alongside some manual data accumulation &
batch processing

5. @avoguru
And Integration was Hard !
5
Lack of Open Standard Protocols
Lack of light weight data formats
Expensive to build adapters - Lack of frameworks
Large development cycles
Not accepted as a problem

6. @avoguru
Integration boom
6
Thanks to
Gregor Hohpe

7. @avoguru
Integration boom
7
This gave rise to:
SOA
Middleware
MQ’s
ESB’s
API Gateways
Adapters/Connectors
All leading to
Integration Pipes (Dumb pipes)

8. @avoguru
8
What improved ?
Overall
architectural
governance
Rise of patterns
organisation wide,
better governance
end to end, leading
to manageable data
flows across the
board
More connected
applications
Giving rise to
automated event
driven data
flows/workflows
supporting
operationally critical
workloads
Faster
development
cycle
Buy Vs Build -
Adapters &
Connectors
everywhere.
Promoted reusability,
Community driven
adapters, all the way
upto the so called
‘Citizen Integrators’
Lesser tech debts
Rise of frameworks,
instead of
reinventing the
wheel for every
integration problem
in every industry

9. Flaws
Traditional Integration Products
can’t meet business demands
anymore - Why ?
9

10. @avoguru
In Summary
10
We focused too much on
improving developer
productivity
& lost focus on fixing the
underlying data storage &
processing bottlenecks.

11. @avoguru
Summary of flaws
11
#1 Increase in Point to Point integrations
(failed promise - never went away)

12. @avoguru
Summary of flaws
12
#2 That's because ‘Integration is tightly coupled’
i.e:
● Incomplete EDA -
Events are always tied
to a specific destination
● Performance is tied to
source/destination
systems

13. @avoguru
Summary of flaws
13
#3 It lacked strong persistence
(heavy reliance on traditional storage mechanisms eg: Databases, File stores)
Integration Pipeline
(including ESB’s & MQ’s)
Database
Filestores (Filesystem,
Object stores, HDFS, etc)
Data processing is not my job -
Because I can't !
Accumulate your data here, so we
can process it.
Real time data is passive from this point
onwards. A Lost opportunity !!!
- World of dumb pipes - Custodians of data @ rest
& Dumping all events in a large file journal != Strong persistence

14. @avoguru
Summary of flaws
14
#4 Requires a multi tier architecture for simple integrations

15. @avoguru
Summary of flaws
15
#5 No data reuse - New integrations everytime
(Heavy reliance on the producer in case of failures/replay)
A B
Queue
C
D
For every new consumer,
you need to run a brand
new integration project

16. @avoguru
Summary of flaws
#6 - Extremely difficult to scale
There is always a rush to deliver the message
MQs & ESBs
16

17. @avoguru
All led to.. A Giant Mess
17

18. We basically forgot dumb pipes were just
workarounds, they are not standards.
It fades away, if you fix the data
persistence problems.
18
@avoguru

19. Rise of Apache Kafka
19

20. @avoguru
Evolution of Kafka
It's a technology used by Data
people
It's used only for high volume
use cases
It is another MQ
It's used only for Data Lake
ingestions
2006 - 2016 2016 - Today
Foundational technology
to build operationally
critical event driven
applications
Central nervous system
that brings application &
data teams together
20

21. Messaging reimagined as a 1st class data
system
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams

22. Writers
Kafka
cluster
Readers
@avoguru

23. Scalability of a Filesystem
• Hundreds of MB/s throughput
• Many TB per server
• Commodity hardware
@avoguru

24. Guarantees of a Database
• Strict ordering
• Persistence
@avoguru

25. Rewind & Replay
Reset to any point in the shared narrative
@avoguru

26. Distributed by Design
• Replication
• Fault Tolerance
• Partitioning
• Elastic Scaling
@avoguru

27. Producing to Kafka
Time
@avoguru

28. Consuming from Kafka
Time
C C
C
@avoguru

29. STREAM
PROCESSING
Create and store
materialized views
Filter
Transformation
in-flight
@avoguru

30. Messaging reimagined as a 1st class data
system
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams

31. 31
Traditional Integration Apache Kafka led Integration
Key benefits
#1 - Simplified architecture
@avoguru

32. 32
Key benefits
#2 - Democratise data / Promotes reuse & replay
@avoguru

33. 33
Key benefits
#3 - No more dumb pipes
Don't have to store data elsewhere, just for the sake of processing it
@avoguru

34. 34
Key benefits
#4 Improved feedback cycle
Don't lose the value of time, React to events on time (bring data & apps together)
@avoguru

35. @avoguru
From here
35

36. To here
36
Kafka
Centralize an immutable stream of facts. Decentralize the freedom to act, adapt, and change.

37. Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-time
as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.

38. Future of Integration Technologies
38

39. @avoguru
39
Process
data on the
fly
Process data
at Rest
Batch Real-time
Data
Integration
Technologies
(ETL)
Traditional
Enterprise
Integration
Technologies
Apache Kafka led
Integration
What's needed for Today’s
business desires !

40. @avoguru
Future of Integration landscape
40
PAST Future
MQ
ESB ESB
MQ
ESB
API
MQ
ESB
API
KAFKA
API
KAFKA
ESP
Connectors/
Lean ESB’s
ESB
API
KAFKA
ESP
Connectors
ESP - Event Stream
Processing Microservices

41. Thank you !
Patterns: https://developer.confluent.io/patterns/
@avoguru
https://www.linkedin.com/gnanaguru
41