Apache Kafka is a widely used open-source platform for building real-time data pipelines and streaming applications. It addresses limitations of using databases to handle high volumes of event data by providing a distributed, scalable, and fault-tolerant event streaming platform. Major companies like Royal Bank of Canada and Carnival Cruise Line rely on Kafka's capabilities for applications like fraud detection, digital marketing, and building event-driven systems.

1. APACHE KAFKA
PAST, PRESENT & FUTURE

2. Second most
active and visited
Apache Project
100,000+
organizations
globally
Trillions of
events per day
60% of
Fortune 100
companies
1000 brokers
per cluster

3. It’s all about data!

4. INSIGHTS
USER PRODUCT
DATA SCIENCE
SIGNALS
VALUE
VIRALITY

5. Initial Database Driven Architecture
DATABASE
WEB APP WEB APP

6. REALIZATION 1
STATEEVENT >

7. STATEEVENT >
I changed my job from
LinkedIn to Confluent.
I work at Confluent.

8. JOB CHANGE RECOMMENDATION ENGINE
SEARCH INDEX
EMAIL SERVICE

9. REALIZATION 2
LEVERAGE ALL
DIGITIZED EVENTS

10. NON-TRANSACTIONAL EVENTSTRANSACTIONAL EVENTS
0.1% 99.9%

11. DATABASE IS A MISMATCH
FOR REALIZATION 1 & 2

12. SQL
SQL
SQL
RECOMMENDATION ENGINE
SEARCH INDEX
EMAIL SERVICE
Mismatch 1: No First Class Treatment for Events
DATABASE

13. Mismatch 2: Not Suitable for Volume of All Digitized Events
DATABASE
1000x more volume
NON-TRANSACTIONAL EVENTS
TRANSACTIONAL EVENTS

14. USER
TRACKING
APPLICATION
LOGS
OPERATIONAL
METRICS
KEY/VALUE
STORE
KEY/VALUE
STORE
DATABASE
MICROSERVICES
...HADOOP
LOG
SEARCH
MONITORING
DATA
WAREHOUSE
REC.
ENGINE SEARCH SECURITY EMAIL
SOCIAL
GRAPH

15. LOG
USER
TRACKING
APPLICATION
LOGS
OPERATIONAL
METRICS
KEY/VALUE
STORE
KEY/VALUE
STORE
DATABASE
MICROSERVICES
...HADOOP
LOG
SEARCH
MONITORING
DATA
WAREHOUSE
REC.
ENGINE SEARCH SECURITY EMAIL
SOCIAL
GRAPH

16. USER
TRACKING
APPLICATION
LOGS
OPERATIONAL
METRICS
KEY/VALUE
STORE
KEY/VALUE
STORE
DATABASE
MICROSERVICES
...HADOOP
LOG
SEARCH
MONITORING
DATA
WAREHOUSE
REC.
ENGINE SEARCH SECURITY EMAIL
SOCIAL
GRAPH
LOG
Why not messaging
systems?

17. MESSAGING
USER
TRACKING
APPLICATION
LOGS
OPERATIONAL
METRICS
KEY/VALUE
STORE
KEY/VALUE
STORE
DATABASE
MICROSERVICES
...HADOOP
LOG
SEARCH
MONITORING
DATA
WAREHOUSE
REC.
ENGINE SEARCH SECURITY EMAIL
SOCIAL
GRAPH

18. MESSAGING
USER
TRACKING
APPLICATION
LOGS
OPERATIONAL
METRICS
KEY/VALUE
STORE
KEY/VALUE
STORE
DATABASE
MICROSERVICES
...HADOOP
LOG
SEARCH
MONITORING
DATA
WAREHOUSE
REC.
ENGINE SEARCH SECURITY EMAIL
SOCIAL
GRAPH
Not designed for volume.
Weak persistence.

19. HIGH THROUGHPUT PUB/SUB
PHASE 1 OF KAFKA:

20. Solving Mismatch 1: Persistent Event Log As First Class Citizen
Solving Mismatch 1:
Persistent Event Log As First Class Citizen
DATABASE
0 1 2 3 4 5 6 7 8LOG
READS
WRITES
DESTINATION
SYSTEM A
(TIME = 2)
DESTINATION
SYSTEM B
(TIME = 6)
DATA
SOURCE

21. Solving Mismatch 2: Distributed Architecture
BROKER 1 BROKER 2 BROKER 3 BROKER 4
TOPIC 1-PART 1
TOPIC 2-PART 3
TOPIC 3-PART 1
TOPIC 2-PART 1
TOPIC 5-PART 1
TOPIC 6-PART 1
TOPIC 1-PART 2
TOPIC 2-PART 4
TOPIC 3-PART 2
TOPIC 2-PART 2
TOPIC 5-PART 2
TOPIC 6-PART 2
CONSUMER CONSUMERCONSUMER
PRODUCER PRODUCERPRODUCER
Solving Mismatch 2: Distributed Architecture

22. DATA DEMOCRACY!
BENEFIT OF KAFKA AFTER PHASE 1:

23. KAFKA JOINED APACHE
THANK YOU TO ALL CONTRIBUTORS!

24. BROKER 1 BROKER 2 BROKER 3 BROKER 4
TOPIC 1-PART 1
Intra-Cluster Replication
TOPIC 2-PART 2
TOPIC 2-PART 1
TOPIC 1-PART 2
TOPIC 1-PART 1
TOPIC 2-PART 2
TOPIC 2-PART 1
TOPIC 1-PART 2
TOPIC 1-PART 1
TOPIC 2-PART 2
TOPIC 2-PART 1
TOPIC 1-PART 2

25. Security
ZOOKEEPER 1
ZOOKEEPER 2
ZOOKEEPER 3
ZOOKEEPER ENSEMBLE
EXTERNAL CLIENT 2EXTERNAL CLIENT 1 EXTERNAL CLIENT N
INTERNAL CLIENT 1
INTERNAL CLIENT 2
INTERNAL CLIENT N
BROKER 1
BROKER 2
BROKER N
KAFKA CLUSTER

26. Connect
CONNECTORCONNECTOR

27. Kafka Streams & KSQL & Exactly-Once Semantics
EVENT-DRIVEN MICROSERVICE
EVENT-DRIVEN MICROSERVICE
EVENT-DRIVEN MICROSERVICE
CREATE STREAM fraudulent_payments AS
SELECT * FROM payments
WHERE fraudProbability > 0.8;
val fraudulentPayments: KStream[String, Payment] = builder
.stream[String, Payment](“payments-kafka-topic”)
.filter((_ ,payment) => payment.fraudProbability > 0.8)
fraudulentPayments.to(“fraudulent-payments-topic”)

28. Complete Event Streaming Platform
NON-TRANSACTIONAL EVENTS
CONNECTOR
CONNECTOR
CONNECTOR
EVENT-DRIVEN
MICROSERVICE
EVENT-DRIVEN
MICROSERVICE
KSQL
KSTREAMS
CONNECTOR
CONNECTOR

29. Royal Bank of Canada — Event Driven Banking
30+ Use Cases
50+ Apps
10+ Different lines
of businesses
Lowering anomaly
detection from
weeks to real-time
EVENT STREAMING PLATFORM
...
DATA
WAREHOUSE
DIGITAL
MARKETING
FRAUD SECURITY
CONSUMER
CREDIT
SERVICES
CORPORATE
REAL
ESTATE
INVESTOR
SERVICES
TREASURY
SERVICES
...

30. Carnival Cruise Line

31. Still Important Work Ahead
ZOOKEEPER
TOPIC 1-PART 1
BROKER 1 BROKER 2 BROKER 3 BROKER 4
TOPIC 2-PART 2
TOPIC 2-PART 1
TOPIC 1-PART 2
TOPIC 1-PART 1
TOPIC 2-PART 2
TOPIC 2-PART 1
TOPIC 1-PART 2
TOPIC 1-PART 1
TOPIC 2-PART 2
TOPIC 2-PART 1
TOPIC 1-PART 2
METADATA METADATA METADATA

32. Still Important Work Ahead
TOPIC 1-PART 1
BROKER 1 BROKER 2 BROKER 3 BROKER 4
TOPIC 2-PART 2
TOPIC 2-PART 1
TOPIC 1-PART 2
TOPIC 1-PART 1
TOPIC 2-PART 2
TOPIC 2-PART 1
TOPIC 1-PART 2
TOPIC 1-PART 1
TOPIC 2-PART 2
TOPIC 2-PART 1
TOPIC 1-PART 2
METADATA METADATAMETADATA

33. Success for your business
not only depends on software,
but how you build software.