This document discusses adding iterative processing capabilities to stream processing systems like Apache Flink. It proposes programming model extensions that treat iterative computations as structured loops over windows. Progress would be tracked using progress timestamps rather than watermarks to allow for arbitrary loop structures. Challenges include managing state and cyclic flow control to avoid deadlocks while encouraging iteration completion.

1. Stream Loops on Flink
Reinventing the wheel for the streaming era
Paris Carbone
@FF2018-Berlin
Systems Researcher@KTH/SICS <parisc@kth.se>
Committer@Flink <senorcarbone@apache.org>
1

2. Stream Compute Landmarks
A Timeline
Sliding
Window
Semantics
Task Parallel
Execution
Arbitrary
Computation
Nesting
Out-of-order
Processing
(watermarks)
State Management
-Fault Tolerance
-Reconfiguration
2

3. Has Streaming Subsumed Batch?
short answer: NO
long answer: not YET
• Staged Processing
• Allows Bulk/Delta Iterative Computation
multi-pass
Flink
DataSet
• Continuous Processing
• Allows correct Acyclic Computation
Flink
DataStream
single-pass
?
3

4. Why Iterations Matter
• Iterations are fundamental building blocks for:
• Graph Analysis (PageRank, Conn.Comp., SSSP etc.)
• Machine Learning (Gradient Descent, PCA etc.)
• Transactions (e.g., optimistic concurrency control)
Iterative Processing Primitives are currently not present
in todays’ production-grade stream processing systems.
4

5. Current Challenges
• Data is born and evolves continuously as a data stream
(e.g., user interactions, sensor events, server logs)
5

6. Current Challenges
Day 1 Day 2 Day 3 Day 4
• To run iterative analysis users have manually do the…
• bucketing
• laundry (schedule jobs)
• cleaning (integration)
6

7. Current Challenges
• To run iterative analysis users have manually do the…
• bucketing
• laundry (iterative job scheduling)
• cleaning (integration)
Day 1 Day 2 Day 3 Day 4
7

8. Current Challenges
• To run iterative analysis users have manually do the…
• bucketing
• laundry (iterative job scheduling)
• sorting (model integration)
Day 1 Day 2 Day 3 Day 4
8

9. Current Challenges
• To run iterative analysis users have manually do the…
• bucketing
• laundry (iterative job scheduling)
• sorting (model integration)
Day 1 Day 2 Day 3 Day 4
9

10. A Partially Solved Problem
• Most challenges mentioned are solved and automated for
single-pass aggregation via stream windowing.
How would a distributed iterative
computation look like as a special
type of window aggregation?
window
computation
windows are a natural fit for
declaring iterative computation
10

11. Anatomy of Iterative Computation
Termination (fixpoint/fixed)
Loop Step Function (computation)
Final Result-Solution
Solution (finite state)
11

12. Programming Model
Extensions
Distributed Dataflow
Execution
DataStream Model
StreamML
Stream
Libraries
Core
Stream API
Runtime
Graph
CEP
SQL
Table
• Multi-Pass Window
Aggregations
• Synchrony and
Termination
• Pregel on Windows
Proposed Additions
based on Flink 1.6
12

13. Window Iterations
An extension of existing window aggregation to multi-pass aggregates
13

14. entry step finalizewindow
iterate
OUTIN
R
Loop Context
14

15. PageRank Example
entry
step
finalize
window
iterate
OUT
IN
15

16. PageRank Example
entry
step
finalize
window
iterate
OUT
IN
16

17. PageRank Example
entry
step
finalize
window
iterate
OUT
IN
17

18. PageRank Example
entry
step
finalize
window
iterate
OUT
IN
18

19. PageRank Example
entry
step
finalize
window
iterate
OUT
IN
19

20. Higher-Level Abstractions
• Vertex-Centric Model (Part of experimental Gelly-Streams lib)
Example:
single source
shortest paths
20

21. Enough about what
• We need to examine “how” to implement iterations
executed in parallel and out-of-order
using event-time progress
window
computation
21

22. How Event-Time Progress Works
T
t
progress metric
record
• Every record carries a timestamp t
• t ≥ T
• T can only increment (monotonicity)
T: indication of a low
event-time boundstream
task
22

23. How Event-Time Progress Works
• low watermarks propagate
progress metrics along the
computational graph
• progress metric = minimum
watermark across channels
• works correctly as long as
monotonicity is maintained
23

24. Intuition
Iteration Superstep progress bears similarities to Event-Time progress
1. Can be used to run computation for out-of-order streams.
2. Need for a decentralised mechanism (no coordinator/scheduler).
3. Only relevant to respective parts of the graph (i.e., sources cannot
make use of sink progress)
24

25. Watermarks and
Unstructured Loops
t2 t4t1
W=n
… …t3
• Low watermarking is a very powerful mechanism to measure and
propagate progress of a single progress metric.
• However it breaks when we
have arbitrary cycles
(current Flink iterations)
old watermark
25
which superstep?
which cycle?

26. Proposed Extensions
Distributed Dataflow
Execution
DataStream Model
StreamML
Stream
Libraries
Core
Stream API
Runtime
Graph
CEP
SQL
Table
• Structured Loops
• Progress Timestamps
• Cyclic Flow Control
runtime Additions
based on Flink 1.6
26

27. Structured Loops
IN OUT
structured
loop
global scope
• Structure loops are low level stream graph primitives.
• They can be seen as ‘Iterative Operators’
• Introduce the notion of “structured programming” for data streaming.
• Each structured loop has a scope.
• Each scope operates on its own progress metric.
• Global scope operates on event-time progress (no changes made).
R
27

28. Structured Loops Composition
R
IN OUT
loop scope
global scope
R’
nested loop scope
28
• Encourages compositionally
(arbitrary nesting)

29. Using Progress Timestamps
• Nests progress timestamps
• (e.g., last superstep pT per window Pctx)
• How do we guarantee monotonic progress metrics?
29

30. Structured Loop Embeddings
R
IN OUT
LIN LOUT
LH LT
nest unnest
incr
[32] [0,32] [32]
[0,32]
[1,32]
[1,32]
[31]
term
[ø,32]
User-Defined Loop DAG
[ø,32][ø,32]
• Initializes Scope’s Progress Metric for
both records and watermarks (nest)
• Triggers final output of an iterative operator
• Removes inner progress metric (unnest)
• Increments Progress Metric (incr)
• Forwards Records and Watermarks
• Evaluates and Disseminates Termination (ø)
30

31. Example: Window Iteration
R
IN OUT
LIN LOUT
LH
LT
entry
step
fin
window
split
Managed State
31

32. Further Challenges
• Cyclic Flow Control
• avoid deadlocks
• encourage iteration completion
• State Management Integration
• one more level of indirection (mapvalue per key)
what keeps us busy
32

33. Cyclic Flow Control
R
IN OUT
LIN LOUT
LH
LT
union
entry
step
fin
window
split
Managed State
flow
control
R
IN
• (Consumed Buffers < Threshold): Round-Robin between IN and R
• (Consumed Buffers >= Threshold): Exclusive Ingestion of R
33

34. FLIP-15
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66853132
• Introduces Scoping and Nesting (for Structured Loops)
• Addresses Flow Control Alternatives
• Discusses Computation-agnostic Loop Termination
34

35. Research Team
and Sponsors
• Marius Melzer (TUDresden)
• Asterios Katsifodimos (TUDelft)
• Vasiliki Kalavri (ETH)
• Seif Haridi (KTH)
• Pramod Bhatotia (Un.Edinburgh)
35

36. References
• Scalable and Reliable Data Stream Processing - Paris Carbone (2018)
• Inflight Progress Tracking for Stream Processing Systems with Cyclic
Dataflows - Marius Melzer (2017)
• Naiad: a timely dataflow system - Murray, McSherry et al. (2013)
• The dataflow model: a practical approach to balancing correctness, latency,
and cost in massive-scale, unbounded, out-of-order data processing - Tyler
Akidau et al. (2015)
• Out-of-order processing: a new architecture for high-
performance stream systems - Li, Maier et al. (2008)
• Flexible time management in data stream systems - Srivastava, Widom
(2004)
36

37. Stream Loops on Flink
Reinventing the wheel for the streaming era
Paris Carbone
@FF2018-Berlin
Systems Researcher@KTH/SICS <parisc@kth.se>
Committer@Flink <senorcarbone@apache.org>
37