Kafka Streams at Scale (Deepak Goyal, Walmart Labs) Kafka Summit NYC 2019
Walmart.com generates millions of events per second. At WalmartLabs, I’m working in a team called the Customer Backbone (CBB), where we wanted to upgrade to a platform capable of processing this event volume in real-time and store the state/knowledge of possibly all the Walmart Customers generated by the processing. Kafka streams’ event-driven architecture seemed like the only obvious choice. However, there are a few challenges w.r.t. Walmart’s scale: • the clusters need to be large and the problems thereof. • infinite retention of changelog topics, wasting valuable disk. • slow stand-by task recovery in case of a node failure (changelog topics have GBs of data) • no repartitioning in Kafka Streams. As part of the event-driven development and addressing the challenges above, I’m going to talk about some bold new ideas we developed as features/patches to Kafka Streams to deal with the scale required at Walmart. • Cold Bootstrap: Where in case of a Kafka Streams node failure, how instead of recovering from the change-log topic, we bootstrap the standby from active’s RocksDB using JSch and zero event loss by careful offset management. • Dynamic Repartitioning: We added support for repartitioning in Kafka Streams where state is distributed among the new partitions. We can now elastically scale to any number of partitions and any number of nodes. • Cloud/Rack/AZ aware task assignment: No active and standby tasks of the same partition are assigned to the same rack. • Decreased Partition Assignment Size: With large clusters like ours (>400 nodes and 3 stream threads per node), the size of Partition Assignment of the KS cluster being few 100MBs, it takes a lot of time to settle a rebalance. Key Takeaways: • Basic understanding of Kafka Streams. • Productionizing Kafka Streams at scale. • Using Kafka Streams as Distributed NoSQL DB
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Kafka Streams at Scale (Deepak Goyal, Walmart Labs) Kafka Summit NYC 2019
Similar to Kafka Streams at Scale (Deepak Goyal, Walmart Labs) Kafka Summit NYC 2019 (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
Kafka Streams at Scale (Deepak Goyal, Walmart Labs) Kafka Summit NYC 2019
- 1. Processing multi-million events per second Walmart
- 2. Kafka Streams at Scale Deepak Goyal Customer Backbone Walmart Labs
- 3. Kafka Streams’ Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
- 5. Event Flow kafka cluster app cluster task-0 task-0’ stand-by task-0’’ stand-by change-log topic partition 0’’ partition 0’ partition 0 input topic partition 0’’ partition 0’ partition 0 rocks db rocks db active rocks db Event Flow @walmartlabs
- 6. Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
- 7. app cluster task-0’ stand-by task-0 kafka cluster task-0’ new stand-by change-log topic partition 0’’ partition 0’ partition 0 input topic partition 0’’ partition 0’ partition 0 active rocks db rocks db the source of truth stand-by recovery Default Bootstrap empty rocks db @walmartlabs
- 8. Default Bootstrap ChangeLog topic as a source of truth • Slow stand-by recovery • Log-Compacted change-log topics • Inefficient disk usage @walmartlabs
- 9. app cluster task-0’ stand-by task-0 task-0’ new stand-by kafka cluster change-log topic partition 0’’ partition 0’ partition 0 input topic partition 0’’ partition 0’ partition 0 active rocks db rocks db the source of truth Cold Bootstrap empty rocks db rocks db enhanced stand-by recovery the new source of truth @walmartlabs
- 10. Cold Bootstrap @walmartlabs Active topic as a source of truth • Lightning stand-by recovery • Efficient disk usage • Bootstrap across data centers
- 11. Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
- 12. partition-0 0,4,8 partition-1 1,5,9 partition-2 2,6,10 partition-3 3,7,11 Repartitioning Logic partition-0 0,2,4,6,8,10 partition-1 1,3,5,7,9,11 partition-0 0,2,4,6,8,10 partition-1 1,3,5,7,9,11 partition-2 0,2,4,6,8,10 partition-3 1,3,5,7,9,11 partition-2 0,2,4,6,8,10 partition-3 1,3,5,7,9,11 @walmartlabs
- 13. Dynamic Repartitioning app cluster task-0 active rocks db task-0’ stand-by rocks db task-0’’ or 2 future stand-by rocks db task-2 becomes active rocks db task-2’ new stand-by empty rocks db rocks db @walmartlabs Scaling up from 2 to 4 partitions
- 14. Scaling Lookups Queryable Stand-by AKKA Server (Non Blocking IO) Partition Specific Lookups @walmartlabs
- 15. Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
- 16. AZ/Rack Aware Task Assignment AZ2 AZ3AZ1 task-0 active rocks db task-0’’ stand-by rocks db task-1 active rocks db task-1’ stand-by rocks db task-0’ stand-by rocks db task-1’’ stand-by rocks db StickyTaskAssignor using RACK_ID_CONFIG = “rack.id”; @walmartlabs
- 18. Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
- 19. Enhancements to Streams’ RocksDB •Eliminated Synchronized GETs •Column Family support •Queryable in suspended and restoration state @walmartlabs
- 20. Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
- 21. Large Clusters Rebalance time • Bottleneck: Group Leader Broker • Partition Assignment Info • Compression • Better Encoding • 24x smaller in terms of size @walmartlabs
- 22. Broker Configs Overriding Broker Defaults • message.max.bytes • offsets.load.buffer.size • replica.fetch.max.bytes • socket.request.max.bytes @walmartlabs
- 23. Streams Configs Overriding Streams Defaults • acks • linger.ms • auto.offset.reset • retries • state.cleanup.delay.ms @walmartlabs
- 24. Results @walmartlabs Staging Benchmark Kafka Cluster 17 XL VMs Streams Cluster 100 M VMs Processing Rate 2.3 million events per second
- 25. Up Next • Feature Extraction and Model Inferencing 😎 • Cold Bootstrap from other stand-by 😍 • Cold-Bootstrap and Repartitioning for DSL🧐 • TTL support for RocksDB 🧐 • Merge Operator for RocksJava 😥 • Chaining Kafka Streams Apps 🧐 • Multi Tenancy 🧐 😪 🧐 😵 @walmartlabs
- 26. keep-streaming . . . . . . . . . . . . . . . . . . . We are hiring! @deepak-iiit Walmart
Editor's Notes
- Interactions (search, page views, item clicks etc.) Transaction, Profile changes. Each event tells a story about the customer, what he/she likes, dislikes, gonna buy. Needed a platform capable of processing this event volume and store the knowledge/state of all the Walmart’s online customers.
- Alternatives: Flink, Spark Streaming, Dynomite Kafka streams seemed to be the only obvious choice (library, scalable, fault tolerable, exactly once semantics) But at Walmart’s Scale, even Kafka Streams came with its own bag of challenges. Self Introduction Forked over an year ago, made more than 100 commits to improve its scalability, fault-tolerance, performance and add more features to it. “Distributed DB” But………
- Slow Fault Recovery Scalability: Limited to number of partitions (one on one mapping to input topics) Cloud management and maintenance Rocks Java: Limitations of usage Rocks DB Store Large Clusters: Long rebalance time.
- Single process with embedded RocksDB and akka server to serve keys from DB.
- N partitions App partition zero maps to kafka topic’s zero’th partition No apostrophe -> Active One apostrophe -> Stand-by Each broker can hold multiple partitions Each app instance can hold multiple tasks Recovers from Change-log topic
- Replays entire change-log Recovery time proportional to amount of data in change-log topics
- 100Gbs connection, 100GB data, 8 seconds Unwindowed stores yet windowed changelog topics Lose exactly-once semantics in cross DC boot straps JSch for file transfer
- One on one mapping with input topic. 100 partitions means 100 Actives and 100 Standby Tasks. Lookups
- Happens as a part of restoration phase. Background threads cleaning unrequired key-value pairs.
- Multiplying serving throughput Notes: A slide for partition specific lookups.
- Availability zones / Racks going down or under maintanance
- Adding RACK_ID_CONFIG to Subscription Info and implementing StickyTaskAssignor
- We always expect a few instances to go down. If so rebalance. But if entire AZ goes down, don’t rebalance. Only if above threshold, then do reassignment on rebalance.
- Replacing with Akka Server did not help as long as Gets were synchronized. Talk about locking with state changes of rocksDB Serving Downtime to potentially zero.
- acks( producer) = all linger.ms (producer) = as required but >0 auto.offet.rest(consumer) = earliest retries (streams) = infinite state.cleanup.delay.ms (streams) = sufficiently large
- We never lose state for any customer. Potentially zero serving down-time Going towards zero processing down-time Streaming Engine and Distributed NoSQL DB with exactly once Semantics, high lookup and processing throughput