SlideShare ist ein Scribd-Unternehmen logo
1 von 65
Downloaden Sie, um offline zu lesen
Scotty: Efficient Window Aggregation with
General Stream Slicing
Berlin, October 7-9, 2019
Philipp M. Grulich
Research Associate (TU Berlin)
Jonas Traub
Research Associate (TU Berlin)
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Aggregations in Stream Processing Pipelines
A stream processing pipeline is a series of concurrently running operators.
2
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Aggregations in Stream Processing Pipelines
A stream processing pipeline is a series of concurrently running operators.
Window
Aggregation
2
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Aggregations in Stream Processing Pipelines
A stream processing pipeline is a series of concurrently running operators.
Window
Aggregation
53
2
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Aggregations in Stream Processing Pipelines
A stream processing pipeline is a series of concurrently running operators.
Window
Aggregation
8
2
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Motivation
3
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Motivation
3
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Research Background
Cutty: Aggregate Sharing for User-Defined Windows
P. Carbone, J. Traub, A. Katsifodimos, S. Haridi, V. Markl
ACM International on Conference on Information and Knowledge Management (CIKM2016)
4
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Research Background
Cutty: Aggregate Sharing for User-Defined Windows
P. Carbone, J. Traub, A. Katsifodimos, S. Haridi, V. Markl
ACM International on Conference on Information and Knowledge Management (CIKM2016)
Scotty: Efficient Window Aggregation for out-of-order Stream Processing
J. Traub, P. M. Grulich, A. R. CuĂ©llar, S. Breß, A. Katsifodimos, T. Rabl, V. Markl
IEEE International Conference on Data Engineering (ICDE 2018)
4
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Research Background
Cutty: Aggregate Sharing for User-Defined Windows
P. Carbone, J. Traub, A. Katsifodimos, S. Haridi, V. Markl
ACM International on Conference on Information and Knowledge Management (CIKM2016)
Scotty: Efficient Window Aggregation for out-of-order Stream Processing
J. Traub, P. M. Grulich, A. R. CuĂ©llar, S. Breß, A. Katsifodimos, T. Rabl, V. Markl
IEEE International Conference on Data Engineering (ICDE 2018)
Efficient Window Aggregation with General Stream Slicing
J. Traub, P. M. Grulich, AR. CuĂ©llar, S. Breß, A. Katsifodimos, T. Rabl, V. Markl
International Conference on Extending Database Technology (EDBT 2019; Best Paper Award)
4
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Research Background
Cutty: Aggregate Sharing for User-Defined Windows
P. Carbone, J. Traub, A. Katsifodimos, S. Haridi, V. Markl
ACM International on Conference on Information and Knowledge Management (CIKM2016)
Scotty: Efficient Window Aggregation for out-of-order Stream Processing
J. Traub, P. M. Grulich, A. R. CuĂ©llar, S. Breß, A. Katsifodimos, T. Rabl, V. Markl
IEEE International Conference on Data Engineering (ICDE 2018)
Efficient Window Aggregation with General Stream Slicing
J. Traub, P. M. Grulich, AR. CuĂ©llar, S. Breß, A. Katsifodimos, T. Rabl, V. Markl
International Conference on Extending Database Technology (EDBT 2019; Best Paper Award)
Scotty Window Processor:
Efficent Window Aggregations for Flink, Beam, and Storm
https://github.com/TU-Berlin-DIMA/scotty-window-processor
4
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Stream Slicing Example
5
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Stream Slicing Example
The number of slices depends on the workload.
6
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Stream Slicing Example
7
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Stream Slicing Example
8
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Stream Slicing Example
9
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Stream Slicing Example
10
We store partial aggregates instead of all tuples. => Small memory footprint.
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Stream Slicing Example
11
We assign each tuple to exactly one slice. => O(1) per-tuple complexity.
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Stream Slicing Example
12
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Stream Slicing Example
We require just a few computation steps to calculate final aggregates. => Low latency.
13
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
General Stream Slicing
14
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
General Stream Slicing
Workload
Characteristics
14
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
General Stream Slicing
Workload
Characteristics
Aggregation
Functions
distributive
algebraic
holistic
associativity
cummutativity
invertibility
14
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
General Stream Slicing
Workload
Characteristics
Window
Types
Context Free
Forward Context Free
Forward Context Aware
Aggregation
Functions
distributive
algebraic
holistic
associativity
cummutativity
invertibility
14
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
General Stream Slicing
Workload
Characteristics
Window
Types
Context Free
Forward Context Free
Forward Context Aware
Window
Measures
time
tuple count
arbitrary
Aggregation
Functions
distributive
algebraic
holistic
associativity
cummutativity
invertibility
14
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
General Stream Slicing
Workload
Characteristics
Window
Types
Context Free
Forward Context Free
Forward Context Aware
Stream
Order
in-order
out-of-order
Window
Measures
time
tuple count
arbitrary
Aggregation
Functions
distributive
algebraic
holistic
associativity
cummutativity
invertibility
14
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
General Stream Slicing
Workload
Characteristics
Window
Types
Context Free
Forward Context Free
Forward Context Aware
Stream
Order
in-order
out-of-order
Window
Measures
time
tuple count
arbitrary
Aggregation
Functions
distributive
algebraic
holistic
associativity
cummutativity
invertibility
General Stream Slicing combines generality and efficiency in a single solution.
14
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
Count-based tumbling
window with a length of 5
tuples.
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Count-based tumbling
window with a length of 5
tuples.
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Count-based tumbling
window with a length of 5
tuples.
11 13 12
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
11 13 12
What if the stream is out-of-order?
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
What if the stream is out-of-order?
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
What if the stream is out-of-order?
5
49
Out-of-order Tuple
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
What if the stream is out-of-order?
5
49
Out-of-order Tuple
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
What if the stream is out-of-order?
5
49
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
What if the stream is out-of-order?
5
49
13 12
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
What if the stream is out-of-order?
5
49
13 12
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
What if the stream is out-of-order?
5
49
13 12
5
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
What if the stream is out-of-order?
5
49
13 125 + - 3
5
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
What if the stream is out-of-order?
5
49
13 123 1+ -5 + - 3
5
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Impact of Workload Characteristics
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Tuple Count
15
Event Time
5 12 13 20 35 37 42 46 48 51 52 57 63 64 65
11 13 12
1 2 1 4 3 1 5 2 2 3 6 1 2 2 1
What if the stream is out-of-order?
5
49
13 123 1+ -5 + - 3
5
What if the aggregation function is not invertible?
15
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Scotty Window Processor:
Efficent Window Aggregations
for Flink, Beam, and Storm
https://github.com/TU-Berlin-DIMA/scotty-window-processor
16
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Key-Facts
Features:
17
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Key-Facts
Features:
● One window operator for many systems.
17
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Key-Facts
Features:
● One window operator for many systems.
● High performance window aggregations with stream slicing.
17
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Key-Facts
Features:
● One window operator for many systems.
● High performance window aggregations with stream slicing.
● Scales to thousands of concurrent windows.
17
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Key-Facts
Features:
● One window operator for many systems.
● High performance window aggregations with stream slicing.
● Scales to thousands of concurrent windows.
● Aggregate sharing among multiple window queries.
17
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Key-Facts
Features:
● One window operator for many systems.
● High performance window aggregations with stream slicing.
● Scales to thousands of concurrent windows.
● Aggregate sharing among multiple window queries.
● Adapts to workload characteristics:
○ Window Types
○ Aggregation Functions
○ Window Measures
○ Stream Order
17
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Key-Facts
Features:
● One window operator for many systems.
● High performance window aggregations with stream slicing.
● Scales to thousands of concurrent windows.
● Aggregate sharing among multiple window queries.
● Adapts to workload characteristics:
○ Window Types
○ Aggregation Functions
○ Window Measures
○ Stream Order
Connectors:

more coming soon

17
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Scotty Core
18
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Scotty Core
18
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Scotty Core
18
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Scotty Core
18
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Scotty Core
Scotty adapts to work load characteristics
and combines generality and efficiency in a single solution.
18
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Benchmark
Concurrent Windows with Built-in Window Operator:
● Flink performs well
with a single window
(no overlap; one
bucket at a time)
0
500.000
1.000.000
1.500.000
2.000.000
2.500.000
1 10 20 50 100 500 1000
Flink Storm Flink on Beam
Throughput(Tuples/sec.)
Number of Councurrent Windows
19
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Benchmark
Concurrent Windows with Built-in Window Operator:
● Flink performs well
with a single window
(no overlap; one
bucket at a time)
0
500.000
1.000.000
1.500.000
2.000.000
2.500.000
1 10 20 50 100 500 1000
Flink Storm Flink on Beam
● With overlapping
concurrent windows,
the throughput drops
drastically.
Throughput(Tuples/sec.)
Number of Councurrent Windows
19
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
0
500.000
1.000.000
1.500.000
2.000.000
2.500.000
1 10 20 50 100 500 1000
Flink+Scotty Storm+Scotty Beam+Flink+Scotty
Benchmark
Concurrent Windows with Scotty:
● With Scotty, the throughput
is independent of the
number of concurrent
windows.
20
Throughput(Tuples/sec.)
Number of Councurrent Windows
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Using Scotty on Flink
1. Clone Scotty and install to maven
21
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Using Scotty on Flink
1. Clone Scotty and install to maven
2. Add Scotty to your Flink Project:
21
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Using Scotty on Flink
1. Initialize Scotty Window Operator
22
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Using Scotty on Flink
1. Initialize Scotty Window Operator
2. Add Window Definitions
22
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Using Scotty on Flink
1. Initialize Scotty Window Operator
3. Add Scotty to your Flink Job
2. Add Window Definitions
22
Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing
Acknowledgements: This talk is supported by the Berlin Big Data Center (01IS14013A), the Berlin Center for Machine Learning (01IS18037A), and Software Campus (1-3000473-18TP).
Scotty Window Processor
Scotty Features:
● One window operator for many systems.
● High performance with stream slicing.
● Scales to thousands of concurrent windows.
● Aggregate sharing among multiple window queries.
● Adapts to workload characteristics
tu-berlin-dima.github.io/
scotty-window-processor
Open Source Repository:
23

Weitere Àhnliche Inhalte

Ähnlich wie FlinkForward Berlin 2019 - Scotty: Efficient Window Aggregation with General Stream Slicing

Replacing Academic Journals
Replacing Academic JournalsReplacing Academic Journals
Replacing Academic JournalsBjörn Brembs
 
Costs of the French PWR
Costs of the French PWRCosts of the French PWR
Costs of the French PWRmyatom
 
Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)
Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)
Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)Jonas Traub
 
Sps Conference Essen 2009 Wi Lettenmaier
Sps Conference Essen 2009 Wi LettenmaierSps Conference Essen 2009 Wi Lettenmaier
Sps Conference Essen 2009 Wi LettenmaierCSCP
 
OPAL-RT RT13: Real time simulation of distribution grids
OPAL-RT RT13: Real time simulation of distribution gridsOPAL-RT RT13: Real time simulation of distribution grids
OPAL-RT RT13: Real time simulation of distribution gridsOPAL-RT TECHNOLOGIES
 
From Cloud to Fog: the Tao of IT Infrastructure Decentralization
From Cloud to Fog: the Tao of IT Infrastructure DecentralizationFrom Cloud to Fog: the Tao of IT Infrastructure Decentralization
From Cloud to Fog: the Tao of IT Infrastructure DecentralizationFogGuru MSCA Project
 
Gridforum Juergen Knobloch Grids For Science 20080402
Gridforum Juergen Knobloch Grids For Science 20080402Gridforum Juergen Knobloch Grids For Science 20080402
Gridforum Juergen Knobloch Grids For Science 20080402vrij
 

Ähnlich wie FlinkForward Berlin 2019 - Scotty: Efficient Window Aggregation with General Stream Slicing (8)

Replacing Academic Journals
Replacing Academic JournalsReplacing Academic Journals
Replacing Academic Journals
 
Costs of the French PWR
Costs of the French PWRCosts of the French PWR
Costs of the French PWR
 
Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)
Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)
Efficient Window Aggregation with General Stream Slicing (EDBT 2019, Best Paper)
 
Sps Conference Essen 2009 Wi Lettenmaier
Sps Conference Essen 2009 Wi LettenmaierSps Conference Essen 2009 Wi Lettenmaier
Sps Conference Essen 2009 Wi Lettenmaier
 
Evacuation Modelling in New Zealand the Result of An Online Survey_Crimson Pu...
Evacuation Modelling in New Zealand the Result of An Online Survey_Crimson Pu...Evacuation Modelling in New Zealand the Result of An Online Survey_Crimson Pu...
Evacuation Modelling in New Zealand the Result of An Online Survey_Crimson Pu...
 
OPAL-RT RT13: Real time simulation of distribution grids
OPAL-RT RT13: Real time simulation of distribution gridsOPAL-RT RT13: Real time simulation of distribution grids
OPAL-RT RT13: Real time simulation of distribution grids
 
From Cloud to Fog: the Tao of IT Infrastructure Decentralization
From Cloud to Fog: the Tao of IT Infrastructure DecentralizationFrom Cloud to Fog: the Tao of IT Infrastructure Decentralization
From Cloud to Fog: the Tao of IT Infrastructure Decentralization
 
Gridforum Juergen Knobloch Grids For Science 20080402
Gridforum Juergen Knobloch Grids For Science 20080402Gridforum Juergen Knobloch Grids For Science 20080402
Gridforum Juergen Knobloch Grids For Science 20080402
 

Mehr von Jonas Traub

Definitely not Java! A Hands-on Introduction to Efficient Functional Programm...
Definitely not Java! A Hands-on Introduction to Efficient Functional Programm...Definitely not Java! A Hands-on Introduction to Efficient Functional Programm...
Definitely not Java! A Hands-on Introduction to Efficient Functional Programm...Jonas Traub
 
Efficient Data Stream Processing in the Internet of Things - SoftwareCampus A...
Efficient Data Stream Processing in the Internet of Things - SoftwareCampus A...Efficient Data Stream Processing in the Internet of Things - SoftwareCampus A...
Efficient Data Stream Processing in the Internet of Things - SoftwareCampus A...Jonas Traub
 
Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentat...
Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentat...Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentat...
Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentat...Jonas Traub
 
Database Research at TU Berlin DIMA and DFKI IAM - USA Excursion Slides 2019
Database Research at TU Berlin DIMA and DFKI IAM - USA Excursion Slides 2019Database Research at TU Berlin DIMA and DFKI IAM - USA Excursion Slides 2019
Database Research at TU Berlin DIMA and DFKI IAM - USA Excursion Slides 2019Jonas Traub
 
Resense: Transparent Record and Replay of Sensor Data in the Internet of Thin...
Resense: Transparent Record and Replay of Sensor Data in the Internet of Thin...Resense: Transparent Record and Replay of Sensor Data in the Internet of Thin...
Resense: Transparent Record and Replay of Sensor Data in the Internet of Thin...Jonas Traub
 
Scotty: Efficient Window Aggregation for Out-of-Order Stream Processing
Scotty: Efficient Window Aggregation for Out-of-Order Stream ProcessingScotty: Efficient Window Aggregation for Out-of-Order Stream Processing
Scotty: Efficient Window Aggregation for Out-of-Order Stream ProcessingJonas Traub
 
Efficient SIMD Vectorization for Hashing in OpenCL
Efficient SIMD Vectorization for Hashing in OpenCLEfficient SIMD Vectorization for Hashing in OpenCL
Efficient SIMD Vectorization for Hashing in OpenCLJonas Traub
 
UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...
UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...
UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...Jonas Traub
 
JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...
JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...
JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...Jonas Traub
 
IÂČ: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
IÂČ: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...IÂČ: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
IÂČ: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...Jonas Traub
 
IÂČ: Interactive Real-Time Visualization for Streaming Data
IÂČ: Interactive Real-Time Visualization for Streaming DataIÂČ: Interactive Real-Time Visualization for Streaming Data
IÂČ: Interactive Real-Time Visualization for Streaming DataJonas Traub
 
LWA 2015: The Apache Flink Platform (Poster)
LWA 2015: The Apache Flink Platform (Poster)LWA 2015: The Apache Flink Platform (Poster)
LWA 2015: The Apache Flink Platform (Poster)Jonas Traub
 
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream AnalysisLWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream AnalysisJonas Traub
 

Mehr von Jonas Traub (13)

Definitely not Java! A Hands-on Introduction to Efficient Functional Programm...
Definitely not Java! A Hands-on Introduction to Efficient Functional Programm...Definitely not Java! A Hands-on Introduction to Efficient Functional Programm...
Definitely not Java! A Hands-on Introduction to Efficient Functional Programm...
 
Efficient Data Stream Processing in the Internet of Things - SoftwareCampus A...
Efficient Data Stream Processing in the Internet of Things - SoftwareCampus A...Efficient Data Stream Processing in the Internet of Things - SoftwareCampus A...
Efficient Data Stream Processing in the Internet of Things - SoftwareCampus A...
 
Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentat...
Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentat...Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentat...
Analyzing Efficient Stream Processing on Modern Hardware (VLDB 2019 Presentat...
 
Database Research at TU Berlin DIMA and DFKI IAM - USA Excursion Slides 2019
Database Research at TU Berlin DIMA and DFKI IAM - USA Excursion Slides 2019Database Research at TU Berlin DIMA and DFKI IAM - USA Excursion Slides 2019
Database Research at TU Berlin DIMA and DFKI IAM - USA Excursion Slides 2019
 
Resense: Transparent Record and Replay of Sensor Data in the Internet of Thin...
Resense: Transparent Record and Replay of Sensor Data in the Internet of Thin...Resense: Transparent Record and Replay of Sensor Data in the Internet of Thin...
Resense: Transparent Record and Replay of Sensor Data in the Internet of Thin...
 
Scotty: Efficient Window Aggregation for Out-of-Order Stream Processing
Scotty: Efficient Window Aggregation for Out-of-Order Stream ProcessingScotty: Efficient Window Aggregation for Out-of-Order Stream Processing
Scotty: Efficient Window Aggregation for Out-of-Order Stream Processing
 
Efficient SIMD Vectorization for Hashing in OpenCL
Efficient SIMD Vectorization for Hashing in OpenCLEfficient SIMD Vectorization for Hashing in OpenCL
Efficient SIMD Vectorization for Hashing in OpenCL
 
UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...
UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...
UZH Stream Reasoning Workshop 2018: Optimized On-Demand Data Streaming from S...
 
JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...
JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...
JT@UCSB - On-Demand Data Streaming from Sensor Nodes and A quick overview of ...
 
IÂČ: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
IÂČ: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...IÂČ: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
IÂČ: Interactive Real-Time Visualization for Streaming Data with Apache Flink ...
 
IÂČ: Interactive Real-Time Visualization for Streaming Data
IÂČ: Interactive Real-Time Visualization for Streaming DataIÂČ: Interactive Real-Time Visualization for Streaming Data
IÂČ: Interactive Real-Time Visualization for Streaming Data
 
LWA 2015: The Apache Flink Platform (Poster)
LWA 2015: The Apache Flink Platform (Poster)LWA 2015: The Apache Flink Platform (Poster)
LWA 2015: The Apache Flink Platform (Poster)
 
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream AnalysisLWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
 

KĂŒrzlich hochgeladen

April 2024 - Crypto Market Report's Analysis
April 2024 - Crypto Market Report's AnalysisApril 2024 - Crypto Market Report's Analysis
April 2024 - Crypto Market Report's Analysismanisha194592
 
âž„đŸ” 7737669865 đŸ”â–» Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...
âž„đŸ” 7737669865 đŸ”â–» Thrissur Call-girls in Women Seeking Men  🔝Thrissur🔝   Escor...âž„đŸ” 7737669865 đŸ”â–» Thrissur Call-girls in Women Seeking Men  🔝Thrissur🔝   Escor...
âž„đŸ” 7737669865 đŸ”â–» Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...amitlee9823
 
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...amitlee9823
 
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...amitlee9823
 
Predicting Loan Approval: A Data Science Project
Predicting Loan Approval: A Data Science ProjectPredicting Loan Approval: A Data Science Project
Predicting Loan Approval: A Data Science ProjectBoston Institute of Analytics
 
âž„đŸ” 7737669865 đŸ”â–» Bangalore Call-girls in Women Seeking Men 🔝Bangalore🔝 Esc...
âž„đŸ” 7737669865 đŸ”â–» Bangalore Call-girls in Women Seeking Men  🔝Bangalore🔝   Esc...âž„đŸ” 7737669865 đŸ”â–» Bangalore Call-girls in Women Seeking Men  🔝Bangalore🔝   Esc...
âž„đŸ” 7737669865 đŸ”â–» Bangalore Call-girls in Women Seeking Men 🔝Bangalore🔝 Esc...amitlee9823
 
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...amitlee9823
 
Probability Grade 10 Third Quarter Lessons
Probability Grade 10 Third Quarter LessonsProbability Grade 10 Third Quarter Lessons
Probability Grade 10 Third Quarter LessonsJoseMangaJr1
 
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...amitlee9823
 
BDSM⚡Call Girls in Mandawali Delhi >àŒ’8448380779 Escort Service
BDSM⚡Call Girls in Mandawali Delhi >àŒ’8448380779 Escort ServiceBDSM⚡Call Girls in Mandawali Delhi >àŒ’8448380779 Escort Service
BDSM⚡Call Girls in Mandawali Delhi >àŒ’8448380779 Escort ServiceDelhi Call girls
 
âž„đŸ” 7737669865 đŸ”â–» Dindigul Call-girls in Women Seeking Men 🔝Dindigul🔝 Escor...
âž„đŸ” 7737669865 đŸ”â–» Dindigul Call-girls in Women Seeking Men  🔝Dindigul🔝   Escor...âž„đŸ” 7737669865 đŸ”â–» Dindigul Call-girls in Women Seeking Men  🔝Dindigul🔝   Escor...
âž„đŸ” 7737669865 đŸ”â–» Dindigul Call-girls in Women Seeking Men 🔝Dindigul🔝 Escor...amitlee9823
 
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...Valters Lauzums
 
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...amitlee9823
 
âž„đŸ” 7737669865 đŸ”â–» Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...
âž„đŸ” 7737669865 đŸ”â–» Mathura Call-girls in Women Seeking Men  🔝Mathura🔝   Escorts...âž„đŸ” 7737669865 đŸ”â–» Mathura Call-girls in Women Seeking Men  🔝Mathura🔝   Escorts...
âž„đŸ” 7737669865 đŸ”â–» Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...amitlee9823
 
Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...only4webmaster01
 
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...Delhi Call girls
 

KĂŒrzlich hochgeladen (20)

April 2024 - Crypto Market Report's Analysis
April 2024 - Crypto Market Report's AnalysisApril 2024 - Crypto Market Report's Analysis
April 2024 - Crypto Market Report's Analysis
 
âž„đŸ” 7737669865 đŸ”â–» Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...
âž„đŸ” 7737669865 đŸ”â–» Thrissur Call-girls in Women Seeking Men  🔝Thrissur🔝   Escor...âž„đŸ” 7737669865 đŸ”â–» Thrissur Call-girls in Women Seeking Men  🔝Thrissur🔝   Escor...
âž„đŸ” 7737669865 đŸ”â–» Thrissur Call-girls in Women Seeking Men 🔝Thrissur🔝 Escor...
 
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
 
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
 
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get CytotecAbortion pills in Doha Qatar (+966572737505 ! Get Cytotec
Abortion pills in Doha Qatar (+966572737505 ! Get Cytotec
 
Predicting Loan Approval: A Data Science Project
Predicting Loan Approval: A Data Science ProjectPredicting Loan Approval: A Data Science Project
Predicting Loan Approval: A Data Science Project
 
âž„đŸ” 7737669865 đŸ”â–» Bangalore Call-girls in Women Seeking Men 🔝Bangalore🔝 Esc...
âž„đŸ” 7737669865 đŸ”â–» Bangalore Call-girls in Women Seeking Men  🔝Bangalore🔝   Esc...âž„đŸ” 7737669865 đŸ”â–» Bangalore Call-girls in Women Seeking Men  🔝Bangalore🔝   Esc...
âž„đŸ” 7737669865 đŸ”â–» Bangalore Call-girls in Women Seeking Men 🔝Bangalore🔝 Esc...
 
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
 
Anomaly detection and data imputation within time series
Anomaly detection and data imputation within time seriesAnomaly detection and data imputation within time series
Anomaly detection and data imputation within time series
 
Probability Grade 10 Third Quarter Lessons
Probability Grade 10 Third Quarter LessonsProbability Grade 10 Third Quarter Lessons
Probability Grade 10 Third Quarter Lessons
 
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
 
BDSM⚡Call Girls in Mandawali Delhi >àŒ’8448380779 Escort Service
BDSM⚡Call Girls in Mandawali Delhi >àŒ’8448380779 Escort ServiceBDSM⚡Call Girls in Mandawali Delhi >àŒ’8448380779 Escort Service
BDSM⚡Call Girls in Mandawali Delhi >àŒ’8448380779 Escort Service
 
âž„đŸ” 7737669865 đŸ”â–» Dindigul Call-girls in Women Seeking Men 🔝Dindigul🔝 Escor...
âž„đŸ” 7737669865 đŸ”â–» Dindigul Call-girls in Women Seeking Men  🔝Dindigul🔝   Escor...âž„đŸ” 7737669865 đŸ”â–» Dindigul Call-girls in Women Seeking Men  🔝Dindigul🔝   Escor...
âž„đŸ” 7737669865 đŸ”â–» Dindigul Call-girls in Women Seeking Men 🔝Dindigul🔝 Escor...
 
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
 
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
 
âž„đŸ” 7737669865 đŸ”â–» Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...
âž„đŸ” 7737669865 đŸ”â–» Mathura Call-girls in Women Seeking Men  🔝Mathura🔝   Escorts...âž„đŸ” 7737669865 đŸ”â–» Mathura Call-girls in Women Seeking Men  🔝Mathura🔝   Escorts...
âž„đŸ” 7737669865 đŸ”â–» Mathura Call-girls in Women Seeking Men 🔝Mathura🔝 Escorts...
 
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
 
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
 
Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 9155563397 👗 Top Class Call Girl Service B...
 
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
 

FlinkForward Berlin 2019 - Scotty: Efficient Window Aggregation with General Stream Slicing

  • 1. Scotty: Efficient Window Aggregation with General Stream Slicing Berlin, October 7-9, 2019 Philipp M. Grulich Research Associate (TU Berlin) Jonas Traub Research Associate (TU Berlin)
  • 2. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Aggregations in Stream Processing Pipelines A stream processing pipeline is a series of concurrently running operators. 2
  • 3. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Aggregations in Stream Processing Pipelines A stream processing pipeline is a series of concurrently running operators. Window Aggregation 2
  • 4. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Aggregations in Stream Processing Pipelines A stream processing pipeline is a series of concurrently running operators. Window Aggregation 53 2
  • 5. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Aggregations in Stream Processing Pipelines A stream processing pipeline is a series of concurrently running operators. Window Aggregation 8 2
  • 6. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Motivation 3
  • 7. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Motivation 3
  • 8. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Research Background Cutty: Aggregate Sharing for User-Defined Windows P. Carbone, J. Traub, A. Katsifodimos, S. Haridi, V. Markl ACM International on Conference on Information and Knowledge Management (CIKM2016) 4
  • 9. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Research Background Cutty: Aggregate Sharing for User-Defined Windows P. Carbone, J. Traub, A. Katsifodimos, S. Haridi, V. Markl ACM International on Conference on Information and Knowledge Management (CIKM2016) Scotty: Efficient Window Aggregation for out-of-order Stream Processing J. Traub, P. M. Grulich, A. R. CuĂ©llar, S. Breß, A. Katsifodimos, T. Rabl, V. Markl IEEE International Conference on Data Engineering (ICDE 2018) 4
  • 10. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Research Background Cutty: Aggregate Sharing for User-Defined Windows P. Carbone, J. Traub, A. Katsifodimos, S. Haridi, V. Markl ACM International on Conference on Information and Knowledge Management (CIKM2016) Scotty: Efficient Window Aggregation for out-of-order Stream Processing J. Traub, P. M. Grulich, A. R. CuĂ©llar, S. Breß, A. Katsifodimos, T. Rabl, V. Markl IEEE International Conference on Data Engineering (ICDE 2018) Efficient Window Aggregation with General Stream Slicing J. Traub, P. M. Grulich, AR. CuĂ©llar, S. Breß, A. Katsifodimos, T. Rabl, V. Markl International Conference on Extending Database Technology (EDBT 2019; Best Paper Award) 4
  • 11. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Research Background Cutty: Aggregate Sharing for User-Defined Windows P. Carbone, J. Traub, A. Katsifodimos, S. Haridi, V. Markl ACM International on Conference on Information and Knowledge Management (CIKM2016) Scotty: Efficient Window Aggregation for out-of-order Stream Processing J. Traub, P. M. Grulich, A. R. CuĂ©llar, S. Breß, A. Katsifodimos, T. Rabl, V. Markl IEEE International Conference on Data Engineering (ICDE 2018) Efficient Window Aggregation with General Stream Slicing J. Traub, P. M. Grulich, AR. CuĂ©llar, S. Breß, A. Katsifodimos, T. Rabl, V. Markl International Conference on Extending Database Technology (EDBT 2019; Best Paper Award) Scotty Window Processor: Efficent Window Aggregations for Flink, Beam, and Storm https://github.com/TU-Berlin-DIMA/scotty-window-processor 4
  • 12. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Stream Slicing Example 5
  • 13. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Stream Slicing Example The number of slices depends on the workload. 6
  • 14. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Stream Slicing Example 7
  • 15. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Stream Slicing Example 8
  • 16. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Stream Slicing Example 9
  • 17. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Stream Slicing Example 10 We store partial aggregates instead of all tuples. => Small memory footprint.
  • 18. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Stream Slicing Example 11 We assign each tuple to exactly one slice. => O(1) per-tuple complexity.
  • 19. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Stream Slicing Example 12
  • 20. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Stream Slicing Example We require just a few computation steps to calculate final aggregates. => Low latency. 13
  • 21. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing General Stream Slicing 14
  • 22. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing General Stream Slicing Workload Characteristics 14
  • 23. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing General Stream Slicing Workload Characteristics Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 14
  • 24. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing General Stream Slicing Workload Characteristics Window Types Context Free Forward Context Free Forward Context Aware Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 14
  • 25. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing General Stream Slicing Workload Characteristics Window Types Context Free Forward Context Free Forward Context Aware Window Measures time tuple count arbitrary Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 14
  • 26. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing General Stream Slicing Workload Characteristics Window Types Context Free Forward Context Free Forward Context Aware Stream Order in-order out-of-order Window Measures time tuple count arbitrary Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility 14
  • 27. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing General Stream Slicing Workload Characteristics Window Types Context Free Forward Context Free Forward Context Aware Stream Order in-order out-of-order Window Measures time tuple count arbitrary Aggregation Functions distributive algebraic holistic associativity cummutativity invertibility General Stream Slicing combines generality and efficiency in a single solution. 14
  • 28. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 15
  • 29. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 15
  • 30. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 Count-based tumbling window with a length of 5 tuples. 15
  • 31. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Count-based tumbling window with a length of 5 tuples. 15
  • 32. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Count-based tumbling window with a length of 5 tuples. 11 13 12 15
  • 33. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 11 13 12 What if the stream is out-of-order? 15
  • 34. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 15
  • 35. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 5 49 Out-of-order Tuple 15
  • 36. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 5 49 Out-of-order Tuple 15
  • 37. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 5 49 15
  • 38. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 What if the stream is out-of-order? 5 49 13 12 15
  • 39. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 12 15
  • 40. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 12 5 15
  • 41. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 125 + - 3 5 15
  • 42. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 123 1+ -5 + - 3 5 15
  • 43. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Impact of Workload Characteristics 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Tuple Count 15 Event Time 5 12 13 20 35 37 42 46 48 51 52 57 63 64 65 11 13 12 1 2 1 4 3 1 5 2 2 3 6 1 2 2 1 What if the stream is out-of-order? 5 49 13 123 1+ -5 + - 3 5 What if the aggregation function is not invertible? 15
  • 44. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Scotty Window Processor: Efficent Window Aggregations for Flink, Beam, and Storm https://github.com/TU-Berlin-DIMA/scotty-window-processor 16
  • 45. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Key-Facts Features: 17
  • 46. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Key-Facts Features: ● One window operator for many systems. 17
  • 47. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Key-Facts Features: ● One window operator for many systems. ● High performance window aggregations with stream slicing. 17
  • 48. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Key-Facts Features: ● One window operator for many systems. ● High performance window aggregations with stream slicing. ● Scales to thousands of concurrent windows. 17
  • 49. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Key-Facts Features: ● One window operator for many systems. ● High performance window aggregations with stream slicing. ● Scales to thousands of concurrent windows. ● Aggregate sharing among multiple window queries. 17
  • 50. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Key-Facts Features: ● One window operator for many systems. ● High performance window aggregations with stream slicing. ● Scales to thousands of concurrent windows. ● Aggregate sharing among multiple window queries. ● Adapts to workload characteristics: ○ Window Types ○ Aggregation Functions ○ Window Measures ○ Stream Order 17
  • 51. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Key-Facts Features: ● One window operator for many systems. ● High performance window aggregations with stream slicing. ● Scales to thousands of concurrent windows. ● Aggregate sharing among multiple window queries. ● Adapts to workload characteristics: ○ Window Types ○ Aggregation Functions ○ Window Measures ○ Stream Order Connectors: 
more coming soon
 17
  • 52. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Scotty Core 18
  • 53. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Scotty Core 18
  • 54. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Scotty Core 18
  • 55. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Scotty Core 18
  • 56. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Scotty Core Scotty adapts to work load characteristics and combines generality and efficiency in a single solution. 18
  • 57. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Benchmark Concurrent Windows with Built-in Window Operator: ● Flink performs well with a single window (no overlap; one bucket at a time) 0 500.000 1.000.000 1.500.000 2.000.000 2.500.000 1 10 20 50 100 500 1000 Flink Storm Flink on Beam Throughput(Tuples/sec.) Number of Councurrent Windows 19
  • 58. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Benchmark Concurrent Windows with Built-in Window Operator: ● Flink performs well with a single window (no overlap; one bucket at a time) 0 500.000 1.000.000 1.500.000 2.000.000 2.500.000 1 10 20 50 100 500 1000 Flink Storm Flink on Beam ● With overlapping concurrent windows, the throughput drops drastically. Throughput(Tuples/sec.) Number of Councurrent Windows 19
  • 59. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing 0 500.000 1.000.000 1.500.000 2.000.000 2.500.000 1 10 20 50 100 500 1000 Flink+Scotty Storm+Scotty Beam+Flink+Scotty Benchmark Concurrent Windows with Scotty: ● With Scotty, the throughput is independent of the number of concurrent windows. 20 Throughput(Tuples/sec.) Number of Councurrent Windows
  • 60. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Using Scotty on Flink 1. Clone Scotty and install to maven 21
  • 61. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Using Scotty on Flink 1. Clone Scotty and install to maven 2. Add Scotty to your Flink Project: 21
  • 62. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Using Scotty on Flink 1. Initialize Scotty Window Operator 22
  • 63. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Using Scotty on Flink 1. Initialize Scotty Window Operator 2. Add Window Definitions 22
  • 64. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Using Scotty on Flink 1. Initialize Scotty Window Operator 3. Add Scotty to your Flink Job 2. Add Window Definitions 22
  • 65. Jonas Traub (TU Berlin), Philipp M. Grulich (TU Berlin) - Efficient Window Aggregation with Stream Slicing Acknowledgements: This talk is supported by the Berlin Big Data Center (01IS14013A), the Berlin Center for Machine Learning (01IS18037A), and Software Campus (1-3000473-18TP). Scotty Window Processor Scotty Features: ● One window operator for many systems. ● High performance with stream slicing. ● Scales to thousands of concurrent windows. ● Aggregate sharing among multiple window queries. ● Adapts to workload characteristics tu-berlin-dima.github.io/ scotty-window-processor Open Source Repository: 23