Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Kafka Streams at Scale (Deepak Goyal, Walmart Labs) Kafka Summit NYC 2019

388 Aufrufe

Veröffentlicht am

Walmart.com generates millions of events per second. At WalmartLabs, I’m working in a team called the Customer Backbone (CBB), where we wanted to upgrade to a platform capable of processing this event volume in real-time and store the state/knowledge of possibly all the Walmart Customers generated by the processing. Kafka streams’ event-driven architecture seemed like the only obvious choice.

However, there are a few challenges w.r.t. Walmart’s scale:
• the clusters need to be large and the problems thereof.
• infinite retention of changelog topics, wasting valuable disk.
• slow stand-by task recovery in case of a node failure (changelog topics have GBs of data)
• no repartitioning in Kafka Streams.

As part of the event-driven development and addressing the challenges above, I’m going to talk about some bold new ideas we developed as features/patches to Kafka Streams to deal with the scale required at Walmart.
• Cold Bootstrap: Where in case of a Kafka Streams node failure, how instead of recovering from the change-log topic, we bootstrap the standby from active’s RocksDB using JSch and zero event loss by careful offset management.
• Dynamic Repartitioning: We added support for repartitioning in Kafka Streams where state is distributed among the new partitions. We can now elastically scale to any number of partitions and any number of nodes.
• Cloud/Rack/AZ aware task assignment: No active and standby tasks of the same partition are assigned to the same rack.
• Decreased Partition Assignment Size: With large clusters like ours (>400 nodes and 3 stream threads per node), the size of Partition Assignment of the KS cluster being few 100MBs, it takes a lot of time to settle a rebalance.

Key Takeaways:
• Basic understanding of Kafka Streams.
• Productionizing Kafka Streams at scale.
• Using Kafka Streams as Distributed NoSQL DB

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Kafka Streams at Scale (Deepak Goyal, Walmart Labs) Kafka Summit NYC 2019

  1. 1. Processing multi-million events per second Walmart
  2. 2. Kafka Streams at Scale Deepak Goyal Customer Backbone Walmart Labs
  3. 3. Kafka Streams’ Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
  4. 4. App Instance Kafka Streams Task task consumer processor rocks db producer akka server task consumer processor rocks db producer akka server task consumer processor rocks db producer akka server @walmartlabs Kafka Streams Instance
  5. 5. Event Flow kafka cluster app cluster task-0 task-0’ stand-by task-0’’ stand-by change-log topic partition 0’’ partition 0’ partition 0 input topic partition 0’’ partition 0’ partition 0 rocks db rocks db active rocks db Event Flow @walmartlabs
  6. 6. Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
  7. 7. app cluster task-0’ stand-by task-0 kafka cluster task-0’ new stand-by change-log topic partition 0’’ partition 0’ partition 0 input topic partition 0’’ partition 0’ partition 0 active rocks db rocks db the source of truth stand-by recovery Default Bootstrap empty rocks db @walmartlabs
  8. 8. Default Bootstrap ChangeLog topic as a source of truth • Slow stand-by recovery • Log-Compacted change-log topics • Inefficient disk usage @walmartlabs
  9. 9. app cluster task-0’ stand-by task-0 task-0’ new stand-by kafka cluster change-log topic partition 0’’ partition 0’ partition 0 input topic partition 0’’ partition 0’ partition 0 active rocks db rocks db the source of truth Cold Bootstrap empty rocks db rocks db enhanced stand-by recovery the new source of truth @walmartlabs
  10. 10. Cold Bootstrap @walmartlabs Active topic as a source of truth • Lightning stand-by recovery • Efficient disk usage • Bootstrap across data centers
  11. 11. Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
  12. 12. partition-0 0,4,8 partition-1 1,5,9 partition-2 2,6,10 partition-3 3,7,11 Repartitioning Logic partition-0 0,2,4,6,8,10 partition-1 1,3,5,7,9,11 partition-0 0,2,4,6,8,10 partition-1 1,3,5,7,9,11 partition-2 0,2,4,6,8,10 partition-3 1,3,5,7,9,11 partition-2 0,2,4,6,8,10 partition-3 1,3,5,7,9,11 @walmartlabs
  13. 13. Dynamic Repartitioning app cluster task-0 active rocks db task-0’ stand-by rocks db task-0’’ or 2 future stand-by rocks db task-2 becomes active rocks db task-2’ new stand-by empty rocks db rocks db @walmartlabs Scaling up from 2 to 4 partitions
  14. 14. Scaling Lookups Queryable Stand-by AKKA Server (Non Blocking IO) Partition Specific Lookups @walmartlabs
  15. 15. Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
  16. 16. AZ/Rack Aware Task Assignment AZ2 AZ3AZ1 task-0 active rocks db task-0’’ stand-by rocks db task-1 active rocks db task-1’ stand-by rocks db task-0’ stand-by rocks db task-1’’ stand-by rocks db StickyTaskAssignor using RACK_ID_CONFIG = “rack.id”; @walmartlabs
  17. 17. Cloud Maintenance AZ3AZ1 task-0 active rocks db task-0’’ stand-by rocks db AZ2 task-0’ stand-by rocks db task-1 active rocks db task-1’ stand-by rocks db task-1task-1’’ stand-by rocks db becomes active @walmartlabs StickyTaskAssignor (ClusterSizeThreshold>=4) AZ2 task-0’ stand-by rocks db task-1’ stand-by rocks db
  18. 18. Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
  19. 19. Enhancements to Streams’ RocksDB •Eliminated Synchronized GETs •Column Family support •Queryable in suspended and restoration state @walmartlabs
  20. 20. Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. RocksDB 5. Large Clusters @walmartlabs
  21. 21. Large Clusters Rebalance time • Bottleneck: Group Leader Broker • Partition Assignment Info • Compression • Better Encoding • 24x smaller in terms of size @walmartlabs
  22. 22. Broker Configs Overriding Broker Defaults • message.max.bytes • offsets.load.buffer.size • replica.fetch.max.bytes • socket.request.max.bytes @walmartlabs
  23. 23. Streams Configs Overriding Streams Defaults • acks • linger.ms • auto.offset.reset • retries • state.cleanup.delay.ms @walmartlabs
  24. 24. Results @walmartlabs Staging Benchmark Kafka Cluster 17 XL VMs Streams Cluster 100 M VMs Processing Rate 2.3 million events per second
  25. 25. Up Next • Feature Extraction and Model Inferencing 😎 • Cold Bootstrap from other stand-by 😍 • Cold-Bootstrap and Repartitioning for DSL🧐 • TTL support for RocksDB 🧐 • Merge Operator for RocksJava 😥 • Chaining Kafka Streams Apps 🧐 • Multi Tenancy 🧐 😪 🧐 😵 @walmartlabs
  26. 26. keep-streaming . . . . . . . . . . . . . . . . . . . We are hiring! @deepak-iiit Walmart

×