Tech Talk @ Google on Flink Fault Tolerance and HA
•
7 gefällt mir•1,940 views
A technical review and examples on the core of the fault tolerance architecture of Flink.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Tech Talk @ Google on Flink Fault Tolerance and HA
Ähnlich wie Tech Talk @ Google on Flink Fault Tolerance and HA (20)
Mehr von Paris Carbone
Mehr von Paris Carbone (11)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Tech Talk @ Google on Flink Fault Tolerance and HA
- 1. Apache Flink Streaming Resiliency and Consistency Paris Carbone - PhD Candidate KTH Royal Institute of Technology <parisc@kth.se, senorcarbone@apache.org> 1 Technical Talk @ Google
- 2. Overview • The Flink Execution Engine Architecture • Exactly-once-processing challenges • Using Snapshots for Recovery • The ABS Algorithm for DAGs and Cyclic Dataflows • Recovery, Cost and Performance • Job Manager High Availability 2
- 3. Current Focus 3 Streaming APIBatch API Flink Optimiser Flink Runtime Table ML Gelly ML Gelly StateManagement
- 4. Executing Pipelines • Streaming pipelines translate into job graphs 4 Flink Runtime Flink Job Graph Builder/Optimiser Flink ClientUser Pipeline Job Graph
- 5. Unbounded Data Processing Architectures 5 (Hadoop, Spark) (Spark Streaming) 1) Streaming (Distributed Data Flow) LONG-LIVED TASK EXECUTION STATE IS KEPT INSIDE TASKS 2) Micro-Batch
- 6. The Flink Runtime Engine • Tasks run operator logic in a pipelined fashion • They are scheduled among workers • State is kept within tasks 6 LONG-LIVED TASK EXECUTION Job Manager • scheduling • monitoring
- 7. Task Failures • Task failures are guaranteed to occur • We need to make them transparent • Can we simply recover a task from scratch? 7 task state consistent state no duplicatesno loss
- 8. Record Acknowledgements • We can monitor the consumption of every event • It guarantees no loss • Duplicates and inconsistent state are not handled 8 task state consistent state no duplicatesno loss bookkeeping
- 9. Atomic transactions per update • It guarantees no loss and consistent state. • No duplication can be achieved e.g. with bloom filters or caching • Fine grained recovery • Non-constant load in external storage can lead to unsustainable execution 9 task state consistent state no duplicatesno loss state state state Distributed Storage Trident
- 10. Lessons Learned from Batch 10 batch-1batch-2 • If a batch computation fails, simply repeat computation as a transaction • Transaction rate is constant • Can we apply these principles to a true streaming execution?
- 12. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution”
- 13. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4
- 14. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4
- 15. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4 consistent cut
- 16. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4 consistent cut
- 17. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4 consistent cut causality violation
- 18. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4 consistent cut inconsistent cut causality violation
- 19. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4 consistent cut inconsistent cut causality violation Idea: We can resume from a snapshot that defines a consistent cut
- 20. Distributed Snapshots 11 Assumptions • repeatable sources • reliable FIFO channels “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4 consistent cut inconsistent cut causality violation Idea: We can resume from a snapshot that defines a consistent cut
- 25. Distributed Snapshots 12 t2t1 snap - t1 snap - t2
- 26. Distributed Snapshots 12 t2t1 snap - t1 snap - t2
- 27. Distributed Snapshots 12 t3t2t1 snap - t1 snap - t2
- 28. Distributed Snapshots 12 t3t2t1 reset from snap t2 snap - t1 snap - t2
- 29. Distributed Snapshots 12 t3t2t1 reset from snap t2 snap - t1 snap - t2 Assumptions • repeatable sources • reliable FIFO channels
- 30. Taking Snapshots 13 t2t1 execution snapshots Initial approach (see Naiad) • Pause execution on t1,t2,.. • Collect state • Restore execution
- 31. Lamport On the Rescue 14 “The global-state-detection algorithm is to be superimposed on the underlying computation: it must run concurrently with, but not alter, this underlying computation”
- 32. Chandy-Lamport Snapshots 15 p1 p2 p1 Using Markers/Barriers • Triggers Snapshots • Separates preshot-postshot events • Leads to a consistent execution cut markers propagate the snapshot execution under continuous ingestion markers
- 33. Asynchronous Snapshots 16 t2t1 snap - t1 snap - t2 snapshotting snapshotting propagating markers
- 51. Asynchronous Barrier Snapshotting Benefits 27 • Taking advantage of the execution graph structure • No records in transit included in the snapshot • Aligning has lower impact than halting
- 58. Snapshots on Cyclic Dataflows • Checkpointing should eventually terminate • Records in transit within loops should be included in the checkpoint
- 67. Implementation in Flink 38 • Coordinator/Driver Actor per job in JM • sends periodic markers to sources • collects acknowledgements with state handles and registers complete snapshots • injects references to latest consistent state handles and back-edge records in pending execution graph tasks • Tasks • snapshot state transparently in given backend upon receiving a barrier and send ack to coordinator • propagate barriers further like normal records
- 68. Recovery 39 • Full rescheduling of the execution graph from the latest checkpoint - simple, no further modifications • Partial rescheduling of the execution graph for all upstream dependencies - trickier, duplicate elimination should be guaranteed
- 70. Node Failures 41 Job Manager Task Manager Task Manager Task Manager task task task Client job graph optimiser JM State • Pending Execution Graphs • Snapshots (State Handles)
- 71. Node Failures 41 Job Manager Task Manager Task Manager Task Manager task task task Client job graph optimiser JM State • Pending Execution Graphs • Snapshots (State Handles)
- 72. Node Failures 41 Job Manager Task Manager Task Manager Task Manager task task task task Client job graph optimiser JM State • Pending Execution Graphs • Snapshots (State Handles)
- 73. Node Failures 41 Job Manager Task Manager Task Manager Task Manager task task task task Client job graph optimiser JM State • Pending Execution Graphs • Snapshots (State Handles)
- 74. Node Failures 41 Job Manager Task Manager Task Manager Task Manager task task task task Client job graph optimiser JM State • Pending Execution Graphs • Snapshots (State Handles) Single Point Of Failure
- 75. Passive Standby JM 42 JM JM JM Zookeeper State • Leader JM Address • Pending Execution Graphs • State Snapshots Client
- 77. Eventual Leader Election 43 JM JM JMt0 time
- 78. Eventual Leader Election 43 JM JM JMt0 t1 JM JM JM time
- 79. Eventual Leader Election 43 JM JM JMt0 t1 t2 JM JM JM JM JM JM …recovering time
- 80. Eventual Leader Election 43 JM JM JMt0 t1 t2 t3 JM JM JM JM JM JM …recovering JM JMJM time
- 81. Scheduling Jobs 44 Job Manager Task Manager Task Manager Task Manager Client optimiser Zookeeper
- 86. Scheduling Jobs 44 Job Manager Task Manager Task Manager Task Manager task task task task Client optimiser Zookeeperjob-1 execution graph task blocking write in Zookeeper
- 87. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper
- 88. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper Checkpointing states
- 89. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper Checkpointing states
- 90. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper state handle Checkpointing states
- 91. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper state handle Checkpointing states
- 92. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper state handle Checkpointing states
- 93. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper state handle Checkpointing states
- 94. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper state handle Checkpointing states
- 95. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper global snapshot state handle Checkpointing states
- 96. HA in Zookeeper • Task Managers query ZK for the current leader before connecting (with lease) • Upon standby failover restart pending execution graphs • Inject state handles from the last global snapshot
- 97. Summary • The Flink execution monitors long running tasks • Establishing exactly-once-processing guarantees with snapshots can be achieved without halting the execution • The ABS algorithm persists the minimal state at a low cost • Master HA is achieved through Zookeeper and passive failover