Kafka is an open-source distributed commit log addressing low latency, high throughput, scalability, fault-tolerance, and disk-based retention. It can be used to build tracking systems, messaging systems, high performance streaming platforms, real-time analysis, audit log….you name it. In our case, it’s been used to build a scalable event-store and messaging platform that stores billions of messages. In this talk, we’re taking a closer look at essential Kafka concepts such as partition rebalancing, offset management, replication, producer/broker request fetching, file segments, etc. to understand what makes Kafka so scalable, resilient, performant and fault tolerant. We will also touch upon Kafka transactions to know what they are and how to leverage them. Last but not the least we will highlight some potential pitfalls that one should watch out when going to production with Kafka.

1. Demystifying Kafka
By :
Nakul Mishra
“If you are not afraid of going deep, you will have no fear of heights either”
#DevoxxBE @nklmish

2. Do you Like Lego?

5. Simplicity is the Key to complexity
#DevoxxPL @nklmish#DevoxxBE @nklmish

6. #DevoxxPL @nklmish
You build complex system
#DevoxxBE @nklmish

7. Starting with a simple idea
#DevoxxPL @nklmish#DevoxxBE @nklmish

8. Traditional Messaging + Distributed System ≠ Best Idea
#DevoxxPL @nklmish#DevoxxBE @nklmish

9. #DevoxxPL @nklmish
Three challenges
#DevoxxBE @nklmish

10. Scalability
#DevoxxPL @nklmish#DevoxxBE @nklmish

11. Throughput
#DevoxxPL @nklmish#DevoxxBE @nklmish

12. Resiliency
#DevoxxPL @nklmish#DevoxxBE @nklmish

13. Why Kafka?
#DevoxxPL @nklmish#DevoxxBE @nklmish

14. #DevoxxPL @nklmish#DevoxxBE @nklmish
• Support both Point-to-point & publish-subscribe (consumer group generalises
this concept).
Kafka - more than a message queue

15. #DevoxxPL @nklmish
Each topic in Kafka has its own journal
#DevoxxBE @nklmish
Producer
Support both Point-to-point & publish-subscribe
Partition-0 Partition-1 Partition-2

16. #DevoxxPL @nklmish#DevoxxBE @nklmish
Support both Point-to-point & publish-subscribe
Rolling log files
Producer
Partition-0 Partition-1 Partition-2

17. #DevoxxPL @nklmish#DevoxxBE @nklmish
Support both Point-to-point & publish-subscribe

18. #DevoxxPL @nklmish#DevoxxBE @nklmish
• Support both Point-to-point & publish-subscribe (consumer group generalises
this concept).
• Highly Scalable, available & durable.
• Kafka Connect interface - pull & push for data.
• Architecture inherits more from storage system like HBase, Cassandra, HDFS
vs traditional message system.
• Stronger ordering guarantees than a traditional messaging system
Kafka - more than a message queue
[1]: New Relic 15 million messages/sec
[2]: LinkedIn 1.1 Trillion messages/day
[3]: Netflix 2 Trillion messages/day at Peak

19. #DevoxxPL @nklmish#DevoxxBE @nklmish
Ordering guarantees, traditional messaging system
R0 R1 R2 R3 …
C0
C1
C2
Parallelconsumption
Server side: Queue retains records in-order on the server
Async delivery
R0, t=1
R2, t=2
R1, t=0
M
essaging
system
solves
this
using
“exclusive
consum
er”

20. Kafka is better than MOM
#DevoxxPL @nklmish#DevoxxBE @nklmish

21. I didn’t mean my Mom ;)
#DevoxxPL @nklmish#DevoxxBE @nklmish

22. #DevoxxPL @nklmish#DevoxxBE @nklmish
MOM vs.
Kafka
Broker Centric Approach Client Centric Approach
Index structures (Btree or Hash Tables) Log structured
Retention impacts performance Designed for Retention
Outrage: Significant slow down
Outrage: won’t cause infrastructure to
slow down significantly
[1]: Large Queue Depth & Performance Problem

23. How Kafka?
#DevoxxPL @nklmish#DevoxxBE @nklmish

24. “Immutability changes everything”
- Pat Helland
#DevoxxPL @nklmish#DevoxxBE @nklmish

25. “If simplicity is the soul of efficiency than Immutability must be the body”
#DevoxxPL @nklmish#DevoxxBE @nklmish

26. #DevoxxPL @nklmish#DevoxxBE @nklmish
Log Structure
Log (append only journal)
Writes (Append only)
Reads (single seek & scan)

27. #DevoxxBE @nklmish

28. #DevoxxPL @nklmish
Scalability - comes from Log
#DevoxxBE @nklmish

29. C1
#DevoxxPL @nklmish
Parallel Consumption - comes from Log
#DevoxxBE @nklmish
Partition-0
Partition-1
Payment topic
Payment-consumer-group

30. C1
#DevoxxPL @nklmish#DevoxxBE @nklmish
Partition-0
Partition-1
Payment topic
Payment-consumer-group
C2
Parallel Consumption - comes from Log

31. C1
#DevoxxPL @nklmish#DevoxxBE @nklmish
Partition-0
Partition-1
Payment topic
Payment-consumer-group
C2
Partition-2
Parallel Consumption - comes from Log

32. #DevoxxPL @nklmish#DevoxxBE @nklmish
• Doesn’t :
• Even import them in JVM
• Buffer messages to user space
• Kernel level IO, copies directly from disk buffer to socket
Java#transferTo
Massive throughput - comes from Log

33. #DevoxxPL @nklmish#DevoxxBE @nklmish
• OS read-ahead cache is impressive.
• Stays nice & warm even if the service is restarted.
Caching - comes from OS

34. Exactly Once Processing, comes from Kafka Transaction
BankTransfer
Service
X
Consume
Y
Produce
#DevoxxPL @nklmish
Exactly once delivery, definitive design
#DevoxxBE @nklmish
KIP-101
KIP-98

35. #DevoxxPL @nklmish
Duplication Source?
#DevoxxBE @nklmish
Producer
Send

36. #DevoxxPL @nklmish
Duplication Source?
#DevoxxBE @nklmish
Producer
Send
Network problem (Acknowledgment error)

37. #DevoxxPL @nklmish
Duplication Source?
#DevoxxBE @nklmish
Idempotent producer , solves this
ProducerResend
dups
Send

38. #DevoxxPL @nklmish
Duplication Source?
#DevoxxBE @nklmish
Consumer
Fetch
ProducerResend
dups
Send

39. #DevoxxPL @nklmish
Duplication Source?
#DevoxxBE @nklmish
Consumer
Fetch
ProducerResend
dups
Send
Consumer crash, (Error committing offset)

40. #DevoxxPL @nklmish
Duplication Source?
#DevoxxBE @nklmish
Consumer
Fetch
ProducerResend
dups
Send
Fetch
dups

41. #DevoxxPL @nklmish
Chandy & Lampot, marker messages
#DevoxxBE @nklmish
Kafka Transaction
Producer
Transaction
Coordinator

42. #DevoxxPL @nklmish
A
Chandy & Lampot, marker messages
#DevoxxBE @nklmish
Kafka Transaction
Producer
Transaction
Coordinator

43. #DevoxxPL @nklmish
A
B
Tx-log
Chandy & Lampot, marker messages
#DevoxxBE @nklmish
Kafka Transaction
Producer
Transaction
Coordinator
(Internal-Kafka-topic), RF =3
init(t.id)

44. #DevoxxPL @nklmish
A
B
Tx-log
Chandy & Lampot, marker messages
#DevoxxBE @nklmish
Kafka Transaction
Producer
Transaction
Coordinator
(Internal-Kafka-topic), RF =3
init(t.id)
beginTransaction()

45. #DevoxxPL @nklmish
A
Data logs
B
C
Tx-log
Chandy & Lampot, marker messages
#DevoxxBE @nklmish
Kafka Transaction
m0
m1
m2
m3
m4
m5
m6
Producer
Transaction
Coordinator
init(t.id)
t.id -> ongoing
(Internal-Kafka-topic), RF =3

46. #DevoxxPL @nklmish
A
D
Data logs
B
C
Tx-log
Chandy & Lampot, marker messages
#DevoxxBE @nklmish
Kafka Transaction
m0
m1
C
m2
m3
m4
C
m5
m6
C
Producer
Transaction
Coordinator
init(t.id)
t.id -> ongoing
t.id -> prepare
t.id -> committed
(Internal-Kafka-topic), RF =3

47. #DevoxxPL @nklmish
A
D
Data logs
B
C
Tx-log
Chandy & Lampor, marker messages
#DevoxxBE @nklmish
Kafka Transaction
m0
m1
C
m2
m3
m4
C
m5
m6
C
Producer
Transaction
Coordinator
init(t.id)
t.id -> ongoing
t.id -> prepare
t.id -> committed
Consum
er:
isolation.level =
read_com
m
itted
(Internal-Kafka-topic), RF =3

48. #DevoxxPL @nklmish#DevoxxBE @nklmish
• Rock solid replication protocol and leader election process.
• Relies on Replication to avoid sync calls!!
Resiliency - comes from Replication
[1]: PacificA

49. #DevoxxPL @nklmish
Filesystem writes
“foo” to disk
Hard disk
#DevoxxBE @nklmish
Write path

50. #DevoxxPL @nklmish
Buffer Cache
Written to
Hard disk
#DevoxxBE @nklmish
Filesystem
writes “Foo”
to disk
Caches
Write path

51. #DevoxxPL @nklmish
Buffer Cache
Filesystem
writes “Foo”
to disk
Written to
Disk drive
controller
cache
Moves to
Caches
Hard disk
#DevoxxBE @nklmish
Write path

52. #DevoxxPL @nklmish
Buffer Cache
Filesystem
writes “Foo”
to disk
Written to
Disk drive
controller
cache
Moves to
Caches
Disk Caches
Moves to
Hard disk
#DevoxxBE @nklmish
Write path

53. #DevoxxPL @nklmish
Buffer Cache
Filesystem
writes “Foo”
to disk
Written to
Disk drive
controller
cache
Moves to
Caches
Hardware-level cache
(write-through or write-back)
Disk Caches
Hard disk
Moves to
#DevoxxBE @nklmish
Write path

54. #DevoxxPL @nklmish
Buffer Cache
Filesystem
writes “Foo”
to disk
Written to
Disk drive
controller
cache
Moves to
Caches
Hardware-level cache
(write-through or write-back)
Disk Caches
Moves to
Finally
written
to disk
Hard disk
#DevoxxBE @nklmish
Write path

55. Replication
P0 P0 P0
Kafka cluster
Leader
A B C
Kafka partition == Replicated Log
#DevoxxBE @nklmish
Producer
P0

56. Replication
P0 P0 P0
Kafka cluster
Leader
A B C
Kafka partition == Replicated Log
Partition: P0
Leader: A
ISR: [B, C]
#DevoxxBE @nklmish
Producer
P0

57. Replication
P0 P0 P0
Kafka cluster
Leader
A B C
Kafka partition == Replicated Log
Partition: P0
Leader: A
ISR: [B, C]
#DevoxxBE @nklmish
Producer
P0
Slow

58. Replication
P0 P0 P0
Kafka cluster
Leader
A B C
Kafka partition == Replicated Log
Partition: P0
Leader: A
ISR: [B]
#DevoxxBE @nklmish
Producer
P0
Slow

59. Replication
P0
P1
P0
P1
P0
P1
Kafka cluster
Leader
Broker-0 Broker-1 Broker-2
Kafka partition == Replicated Log
Partition: P0
Leader: Broker-0
#DevoxxBE @nklmish
Producer
P1

60. Replication
P0
P1
P0
P1
P0
P1
Kafka cluster
Leader
Broker-0 Broker-1 Broker-2
Kafka partition == Replicated Log
Partition: P0
Leader: Broker-0
#DevoxxBE @nklmish
Producer
Partition: P1
Leader: Broker-1

61. Replication
P0
P1
P2
P0
P1
P2
P0
P1
P2
Kafka cluster
Leader
Broker-0 Broker-1 Broker-2
Kafka partition == Replicated Log
Partition: P0
Leader: Broker-0
#DevoxxBE @nklmish
Producer
Partition: P1
Leader: Broker-1
Partition: P2
Leader: Broker-2

62. #DevoxxPL @nklmish
Kafka & Event sourcing?
#DevoxxBE @nklmish

63. #DevoxxPL @nklmish#DevoxxBE @nklmish
Events ordering - send to same partition (aggregate Identifier).
Events replay - Log (replayable)
Data Retention & Durability
Event store & projections - let’s find out
Kafka - for event sourcing ?

64. Event store ?
#DevoxxPL @nklmish#DevoxxBE @nklmish

65. Event Store
#DevoxxPL @nklmish
Time
#DevoxxBE @nklmish

66. Event Store
#DevoxxPL @nklmish
Time
#DevoxxBE @nklmish

67. Event Store
#DevoxxPL @nklmish
Time
#DevoxxBE @nklmish

68. Event Store
#DevoxxPL @nklmish
Time
#DevoxxBE @nklmish

69. Event Store
#DevoxxPL @nklmish
Time
#DevoxxBE @nklmish

70. Event Store
#DevoxxPL @nklmish
validate sequence number
Time
#DevoxxBE @nklmish

71. Event Store - Load aggregate
#DevoxxPL @nklmish
Time
#DevoxxBE @nklmish

72. #DevoxxPL @nklmish
Time
Event Store - Load aggregate
#DevoxxBE @nklmish

73. #DevoxxPL @nklmish
Event Store - aggregate loaded
#DevoxxBE @nklmish

74. What’s missing
#DevoxxPL @nklmish
• Identity & Concurrency Control
• Load efficiently events for specific aggregates
#DevoxxBE @nklmish
}No builtin support in Kafka, but…

75. #DevoxxPL @nklmish
Kafka, a database inside out
Kafka
Commit Log
Stream
Processor
KSQ
L
Kafka
Indexes
Commit log
Query engine
Caching
Traditional database
#DevoxxBE @nklmish

76. #DevoxxPL @nklmish#DevoxxBE @nklmish
• Following options:
• Optimistic concurrency control
• Single write principle
Identity & concurrency control

77. #DevoxxPL @nklmish#DevoxxBE @nklmish
• Kafka Streams
Aggregates & Projections

78. Business logic
Kafka Stream
WalletService
Kafka Streams
#DevoxxBE @nklmish
Bob:+100 Doe:-100 Bob:+200 Bob:+400 Lee:+400 Bob:-200 Jen:+200 Doe:+200 Jen:+100
1. Default state store, rockDB
2. Local writes to event store -> pushed back to Kafka
State store

79. #DevoxxPL @nklmish#DevoxxBE @nklmish
• Be conscious, when setting deletion policy -> (Integer.MAX_VALUE) is not good ;)
• Shared storage, remember neighbours can be chatty.
• Avoid network throttling & instability, enforce Quotas.
• Adding new broker != automatic data partitions assignment for existing topic,
use bin/kafka-reassign-partitions.sh
• Aggregate (Stateful processing)-> use compacted topic -> avoid loading whole
versioned history.
• Slow controlled shutdown, don’t use kill command -> upgrade to Kafka 1.1.0 :)
What we learned in the process?

80. #DevoxxPL @nklmish#DevoxxBE @nklmish
• Kafka is always on
• Designed from ground up for distributed world
• Kafka - a database inside out
• Its fun to play lego
How Kafka helped us go through the 3 challenges?

81. #DevoxxPL @nklmish#DevoxxBE @nklmish