Deep Dive into Apache Kafka
•
48 gefällt mir•8,073 views
In the last few years, Apache Kafka has been used extensively in enterprises for real-time data collecting, delivering, and processing. In this presentation, Jun Rao, Co-founder, Confluent, gives a deep dive on some of the key internals that help make Kafka popular. - Companies like LinkedIn are now sending more than 1 trillion messages per day to Kafka. Learn about the underlying design in Kafka that leads to such high throughput. - Many companies (e.g., financial institutions) are now storing mission critical data in Kafka. Learn how Kafka supports high availability and durability through its built-in replication mechanism. - One common use case of Kafka is for propagating updatable database records. Learn how a unique feature called compaction in Apache Kafka is designed to solve this kind of problem more naturally.
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Deep Dive into Apache Kafka
Ähnlich wie Deep Dive into Apache Kafka (20)
Mehr von confluent
Mehr von confluent (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Deep Dive into Apache Kafka
- 1. Deep Dive into Apache Kafka Jun Rao co-founder of Confluent
- 2. Kafka Usage
- 4. Agenda • High throughput • Reliability and durability • Compacted topic
- 5. Scaled-out Architecture 5 Kafka cluster broker 1 … producer producer producer consumer consumer broker 2 broker n topic partition
- 7. Detailed Log Representation 7 offset 0 - 10000 timestamp index offset index offset 10001 - 20000 offset 20001 - 30000 offset index offset index timestamp index timestamp index
- 8. Message Format offset message length CRC timestamp key length key content value length value content 8 bytes 4 bytes 4 bytes magic byte 1 byte attribute 1 byte 8 bytes 4 bytes varies 4 bytes varies
- 9. Batching and Compression compressed batch 1send() send() send() send() producer async flush poll()compressed batch 2 compressed batch 3 compressed batch 1 compressed batch 2 compressed batch 3 consumerbroker
- 10. Agenda • High throughput • Reliability and durability • Compacted topic
- 11. Kafka Replication • Configurable replication factor • Tolerating f – 1 failures with f replicas • Unlike quorum based replication • Automated failover
- 12. Replicas and Layout • Topic partition has replicas • Replicas spread evenly among brokers topic1-part1 logs broker 1 topic1-part2 logs broker 2 topic2-part2 topic2-part1 logs broker 3 topic1-part1 logs broker 4 topic1-part2 topic2-part2 topic1-part1 topic1-part2 topic2-part1 topic2-part2 topic2-part1
- 13. High Level Data Flow in Replication broker 1 producer leader broker 2 follower broker 3 follower 4 2 2 3 commit ack When producer receives ack Latency Durability on failures acks=0 (no ack) no network delay some data loss acks=1 (wait for leader) 1 network roundtrip a few data loss acks=all (wait for committed) 2 network roundtrips no data loss topic1-part1 topic1-part1 topic1-part1 consumer 1
- 14. Extend to Multiple Partitions Leaders are evenly spread among brokers broker 1 broker 2 topic3-part1 follower broker 3 topic3-part1 follower topic1-part1 producer leader topic1-part1 follower topic1-part1 follower broker 4 topic3-part1 leader producer topic2-part1 producer leader topic2-part1 follower topic2-part1 follower
- 15. In-sync Replicas (ISR) broker 1 producer leader broker 2 follower broker 3 follower 2 2 topic1-part1 topic1-part1 topic1-part1 1 m1 m1 m1 m2 m2 m2 ISR last committed m2, m1 In-sync : replica reads from leader’s log end within replica.lag.time.max.ms
- 16. Follower Failure broker 1 producer leader broker 2 follower broker 3 follower 2 2 topic1-part1 topic1-part1 topic1-part1 1 m1 m1 m1 m2 m2 m2 ISR last committed
- 17. Shrinking ISR broker 1 producer leader broker 2 follower broker 3 2 topic1-part1 topic1-part1 topic1-part1 1 m1 m1 m1 m2 m2 m2 ISR m3 m3 m4 m4last committed m4, m3 follower
- 18. Failed Replica Coming Back broker 1 producer leader broker 2 follower broker 3 2 topic1-part1 topic1-part1 topic1-part1 1 m1 m1 m1 m2 m2 m2 ISR m3 m3 m4 m4last committed m3 2 follower
- 19. Leader Failure broker 1 producer leader broker 2 follower broker 3 2 topic1-part1 topic1-part1 topic1-part1 1 m1 m1 m1 m2 m2 m2 ISR m3 m3 m4 m4last committed m3 2 follower
- 20. Selecting New Leader from ISR broker 1 producer leader broker 2 leader broker 3 2 topic1-part1 topic1-part1 topic1-part1 1 m1 m1 m1 m2 m2 m2 ISR m3 m3 m4 m4last committed m3 follower
- 21. Expanding ISR broker 1 producer leader broker 2 leader broker 3 2 topic1-part1 topic1-part1 topic1-part1 1 m1 m1 m1 m2 m2 m2 ISR m3 m3 m4 m4 last committed m3 follower m4 m5 m5 m5
- 22. m5 Unclean Leader Election broker 1 producer leader ??? broker 2 leader broker 3 2 topic1-part1 topic1-part1 topic1-part1 1 m1 m1 m1 m2 m2 m2 ISR m3 m3 m4 m4last committed m3 follower m4 m5
- 23. m5 Guaranteed Replicas broker 1 producer broker 2 leader broker 3 topic1-part1 topic1-part1 topic1-part1 1 m1 m1 m1 m2 m2 m2 m3 m3 m4 m4 last committed m3 m4 m5 ISR > min.insync.replicas ? m6
- 24. Mission Critical Data • Disable Unclean Leader Election • unclean.leader.election.enable = false • Set replication factor • default.replication.factor = 3 • Set minimum ISRs • min.insync.replicas = 2 • Set producer acks • acks = all 24
- 25. Failure Detection and Controller Flow broker 1 broker 2 topic3-part1 follower broker 3 topic3-part1 follower topic1-part1 controller leader topic1-part1 follower topic1-part1 follower broker 4 topic3-part1 leader topic2-part1 leader topic2-part1 follower topic2-part1 follower new leaders and ISRs
- 26. Agenda • High throughput • Reliability and durability • Compacted topic
- 27. Log Compaction: What is it?
- 28. Use Case product catalog search index item1 “new description”
- 29. Adding a New Index Instance product catalog search index item1 “new description” new search index
- 30. Using a Compacted Topic product catalog search index item1 “new description” new search index set consumer offset to 0
- 31. Log Cleaner Implementation offset 3001 - 4000 firstDirty 1. build map 2. scan and probe map firstDirty offset 2001 - 3000offset 1001 - 2000 offset 4001 - 5000 offset 5001 - 6000 offset 3001 - 5000offset 1001 - 3000 offset 5001 - 6000 key1 3500 key2 3700 key3 4200key1 1500 key last offset reject key4 2100 keep key3 4200 keep
- 32. Cleaning Configs • log.cleaner.min.cleanable.ratio (default 0.5) • dirty/total ratio when log cleaner is triggered • log.cleaner.io.max.bytes.per.second (default infinite) • Max rate cleaning can be done • Can be used for throttling
- 33. Be Careful with Deletes • Delete tombstone modeled as null message • Danger of removing a deleted key too soon • Consumer still assumes the old value with the key • log.cleaner.delete.retention.ms (default 1 day) • “Delete tombstone” removed after that time • Consumer needs to finish consuming the tombstone before that time
- 34. Summary • Apache Kafka is a streaming platform • The storage part supports • High throughput • High availability and durability • Retaining database-like data
- 35. Coming Up Next Date Title Speaker 10/27 Data Integration with Kafka Gwen Shapira 11/17 Demystifying Stream Processing Neha Narkhede 12/1 A Practical Guide To Selecting A Stream Processing Technology Michael Noll 12/15 Streaming in Practice: Putting Apache Kafka in Production Roger Hoover
Hinweis der Redaktion
- It’s widely used and in production at thousands of companies. Let’s walk through the the basics of Kafka and understand how it acts as a streaming platform.
- Now we’ve talked about these two motivations for streams---solving pipline spawl and asynchronous stream processing. It won’t surprise anyone that when I talk about this streaming platform that enables these pipelines and processing I am talking about Apache Kafka.
- New producer api. Be aware of non-java api.
- Put partitions inside topic? Maybe useful to use another picture. 3-tier
- New producer api. Be aware of non-java api.
- Leaders spread over brokers. Another picture. Animated transitioin.
- New producer api. Be aware of non-java api.
- Actually in 0.8.1
- Actually in 0.8.1