12. What is messaging in essence?
• Take a message, keep it safe, make it available to
consumers.
• Track what messages have been consumed
Kafka attacks these problems separately
13. What is a message
broker in essence?
Sender Receiver
Broker
(the log)
14. The log is a simple idea
Messages are added at the end of the log
Just think of the log as a file
Old New
15. Consumers have a position
Sally
is here
George
is here
Fred
is here
Old New
Scan Scan
Scan
20. The problem:
If you built a messaging
system for internet scale,
what would it look like?
21. Shard data to get scalability
Messages are sent to
different partitions
Producer (1) Producer (2) Producer (3)
Cluster of
machines Partitions live on
different machines
22. Replicate to get fault tolerance
replicate
msg
mastership
moves
machines
(1)
(2)
msg
leader
Machine A
Machine A
Machine B
Machine B
23. Kafka goes a step further
A single topic can be spread
over multiple consumers
(4 consuming machines
process a single topic)
38. Compacted Topics
(Tabular View)
Version 3
Version 2
Version 1
Version 2
Version 1
Version 5
Version 4
Version 3
Version 2
Version 1
Version 2
Version 3
Version 5
All versions Latest Key
only