Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Flink Forward San Francisco 2019: The Trade Desk's Year in Flink - Jonathan Miles

217 Aufrufe

Veröffentlicht am

The Trade Desk's Year in Flink
At advertising technology leader, The Trade Desk, we built a data pipeline with three distinct large-scale products using Flink. The keynote gives you a peek into our journey, the lessons learned and offers some hard-won tips from the trenches. Most jobs were surprisingly simple to build. However, we'll deep-dive into one particularly challenging Flink job where we learned how to synchronise/union multiple streams, both in terms of asymmetric throughput and differing lateness/time.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Flink Forward San Francisco 2019: The Trade Desk's Year in Flink - Jonathan Miles

  1. 1. The Trade Desk’s Year with Flink Jonathan Miles
  2. 2. The Crux of (or Intro to) AdTech Publishers Ad Exchange Advertisers and Brands Real-Time Bidding How to be paid for content? How to get value and reach?
  3. 3. The Crux of (or Intro to) AdTech Publishers Ad Exchange Advertisers and Brands Real-Time Bidding How to be paid for content? How to get value and reach?
  4. 4. The Trade Desk Data Systems Bidders Ads Won Vertica MS SQL Data Pipeline Kafka + Flink <Ververica Platform> HBase S3 “INCOMING LOGS” Ads Bid On Batch Reports (Excel / TSV / CSV) Interactive Reports (UI) Transform Data Lake Aggregate ETL Stream Processing
  5. 5. • Apache Flink - High-level/abstract, lower testing surface area - Most of the distributed systems work is done for you - Common sysadmin challenges • Ververica Platform - Orchestration with App Manager on Kubernetes • Ververica Support - When embracing open-source, The Trade Desk needed "enterprise grade“ software (revenues) - Lean on Ververica, allowing us to work on our business - Gateway to Flink community Flink Highlights
  6. 6. Overview of Flink jobs at TTD Data Pipeline Kafka + Flink <Ververica Platform> HBase S3Data Lake Map + Union + Aggregate SQL Map + Transform
  7. 7. • Data Lake - Five jobs, one for each input topic - Scaled independently - Internal Data Science - … also advertising partners • Map + Transform - Two jobs, for different topics - Single input and output to Hbase - More Data Science use cases Anatomy: simpler jobs Data Pipeline HBase S3Data Lake Map + Transform Kafka + Flink <Ververica Platform>
  8. 8. • Map + Aggregate (Real-Time Insights) - Aggregate disparate input topics - Brands can query DB interactively - Stream processing • Reasonable latency • Reduce complexity of management - Synchronise five input topics with • … different data rates • … different time characteristics Anatomy: challenging jobs Data Pipeline Map + Union + Aggregate SQL Kafka + Flink <Ververica Platform>
  9. 9. Anatomy: challenging jobs Data N rows/sec Data Pre-process: - Union - Aggregate Servers Database Stream Processor
  10. 10. Anatomy: challenging jobs Rollup by… Dim1 Dim2 Dim3 Dim4 Dim5 L1 L2 N rows/sec Database Kafka Two Flink applications
  11. 11. Anatomy: challenging jobs Synchronising input streams is hard! Rollup by… Dim1 Dim2 Dim3 Dim4 Dim5 L1 L2 N rows/sec Database Kafka +/- Throughput +/- Time
  12. 12. Anatomy: challenging jobs Beware of hotspots! Synchronising input streams is hard! Rollup by… Dim1 Dim2 Dim3 Dim4 Dim5 L1 L2 N rows/sec Database Kafka
  13. 13. Anatomy: challenging jobs Hotspots… … caused global slow-down during catchup.
  14. 14. Tips: synchronizing input streams is hard Key T I1C1..n I2C1..n I3C1..n I4C1..n I5C1..n K1 T1 Value Value Value Value Value K2 T2 Value Value Value Value Value SQL Table SELECT SUM(I1.C1), SUM(I1.C2)... FROM Input1 I1 JOIN Input2 I2 ON (I2.key=I1.key…) JOIN Input3 I3 ON (I3.key=I1.key…) JOIN Input4 I4 ON (I4.key=I1.key…) JOIN Input5 I5 ON (I5.key=I1.key…) WHERE Input1.key = @key GROUP BY Key, Window Synchronising input streams is hard! L1
  15. 15. Tips: synchronizing input streams is hard THREE CHALLENGES: • Time synchronisation • Throughput synchronisation • Late data Synchronising input streams is hard! L1
  16. 16. Tips: synchronizing input streams is hard Time synchronisation Map +1 hour +/- 15 mins +/- 15 mins +/- 15 mins +/- 15 mins +1 hour Source Source Source Source Source Map Map Map Map Union Window Aggregate Event Time == Slowest Input Buffers to State Store (slowest input)
  17. 17. Tips: synchronizing input streams is hard BoundedOutOfOrdernessTimestampExtractor Time synchronisation (out of order) Map +1 hour +/- 15 mins +/- 15 mins +/- 15 mins +/- 15 mins Source Source Source Source Source Map Map Map Map +1 hour Union Window Aggregate Out of Order == Late Data Dropped records (still slowest)
  18. 18. Tips: synchronizing input streams is hard Time synchronisation (different latency) Map +15 mins Separate DAG +1 hour Source Source Source Source Source Map Map Map Map Union Window Aggregate Union Window Aggregate+1 hour +/- 15 mins +/- 15 mins +/- 15 mins +/- 15 mins (two writes) Less Data in State Store Multiple DAGs == Flexible
  19. 19. Tips: synchronizing input streams is hard THREE CHALLENGES: • Time synchronisation • Throughput synchronisation • Late data Synchronising input streams is hard! L1
  20. 20. Tips: synchronizing input streams is hard Throughput balancing Map >1% rps <1% rps 6% rps 8% rps 84% rps Source Source Source Source Source Map Map Map Map Union Window Aggregate Union Window Aggregate Constipation Unbalanced Reads == Event Time Skew Everything Buffers to State Store
  21. 21. Tips: synchronizing input streams is hard Job Manager Task Manager Task Manager Task Manager Task Manager Task Manager
  22. 22. Tips: synchronizing input streams is hard Job Manager Slot Slot Slot Slot Slot One Slot Per Task Manager Entire Pipeline Inside Every Slot … reading all inputs
  23. 23. Tips: synchronizing input streams is hard Throughput balancing Map Source Source Source Source Source Map Map Map Map Union Window Aggregate Union Window Aggregate Every Slot Has These Two DAGs (e.g.) >1% rps <1% rps 6% rps 8% rps 84% rps 300 instances of every node in the DAG… … leads to some being starved
  24. 24. Tips: synchronizing input streams is hard Slot Sharing Groups Low throughput Medium throughput High throughput 2 slots 14 slots 90 slots 194 slots 300 total slots Window Aggregate 1. Isolation 2. Reads in proportion
  25. 25. Tips: synchronizing input streams is hard Job Manager S S S S S S S S S S S S S S SS S S S S S S S S S S SS S S S S SSS S SS S S S S SSS S SS … ongoing work to make this easier out of the box Cluster of slots, dedicated to parts of DAG
  26. 26. Tips: synchronising input streams is hard Shared Slots (default) Slot Groups Control rate of consumption Even distribution of resource Easy attribution of effects
  27. 27. Tips: synchronizing input streams is hard THREE CHALLENGES: • Time synchronisation • Throughput synchronisation • Late data Synchronising input streams is hard! L1
  28. 28. Tips: synchronizing input streams is hard Map Source Source Source Source Source Map Map Map Map Union Window Aggregate Union Window Aggregate Late Data Late path Late path +15 mins +1 hour Event Time (Processing Time)
  29. 29. • Flink’s flexibility and demanding use-case • Unbalanced time - BoundedOutOfOrdernessTimestampExtractor (+/- 15 minutes) - Separate DAG in same job (+1 hour) - Actual late data shunted to separate path • Unbalanced throughput, Flink won't balance reads for you - TTD: Slot Model → Slot Groups - Future/community: Event time synchronization across sources (FLINK-10886) • Constipation and laxative - Time-based distribution of individual topic - Processing-time eviction • Real-time data is okay, catchup is main problem Tips: synchronizing input streams is hard
  30. 30. • Why was Flink a good choice? - Generalised framework, well defined semantics - Most distributed systems problems under control - Flexibility • CDC project - Generalised/democratised access to snapshots of database tables • What Flink features are we excited about? - Event time synchronization across sources (FLINK-10886) - Ability to take checkpoints during high backpressure (alignment sentinels stuck in same queue as messages) • We can dream… - Generalised solution to hotspots problem Future
  31. 31. • Diagnostic tool, flink-diag.py - GitHub? - Come talk to us? • We’re hiring - Come change the world of advertising with us! • Questions? - Many people from The Trade Desk attending today - Feel free to come talk to us! Plug
  32. 32. Thank you

×