4. Flink 1.1+ ongoing development
4
Session Windows(Stream) SQL
Library
enhancements
Metric
System
Metrics &
Visualization
Dynamic Scaling
Savepoint
compatibility Checkpoints
to savepoints
Connectors in Flink
Stream SQL
Windows
Large state
Maintenance
Fine grained
recovery
Side in-/outputs
Window DSL
Security
Mesos &
others
Dynamic Resource
Management
Authentication
Queryable StateApache Bahir connectors
Operations
Ecosystem
Application
Features
Broader
Audience
5. Flink 1.1+ ongoing development
4
Session Windows(Stream) SQL
Library
enhancements
Metric
System
Metrics &
Visualization
Dynamic Scaling
Savepoint
compatibility Checkpoints
to savepoints
Connectors in Flink
Stream SQL
Windows
Large state
Maintenance
Fine grained
recovery
Side in-/outputs
Window DSL
Security
Mesos &
others
Dynamic Resource
Management
Authentication
Queryable StateApache Bahir connectors
Operations
Ecosystem
Application
Features
Broader
Audience
6. Flink 1.1+ ongoing development
4
Session Windows(Stream) SQL
Library
enhancements
Metric
System
Metrics &
Visualization
Dynamic Scaling
Savepoint
compatibility Checkpoints
to savepoints
Connectors in Flink
Stream SQL
Windows
Large state
Maintenance
Fine grained
recovery
Side in-/outputs
Window DSL
Security
Mesos &
others
Dynamic Resource
Management
Authentication
Queryable StateApache Bahir connectors
Operations
Ecosystem
Application
Features
Broader
Audience
7. Flink 1.1+ ongoing development
4
Session Windows(Stream) SQL
Library
enhancements
Metric
System
Metrics &
Visualization
Dynamic Scaling
Savepoint
compatibility Checkpoints
to savepoints
Connectors in Flink
Stream SQL
Windows
Large state
Maintenance
Fine grained
recovery
Side in-/outputs
Window DSL
Security
Mesos &
others
Dynamic Resource
Management
Authentication
Queryable StateApache Bahir connectors
Operations
Ecosystem
Application
Features
Broader
Audience
8. Flink 1.1+ ongoing development
4
Session Windows(Stream) SQL
Library
enhancements
Metric
System
Metrics &
Visualization
Dynamic Scaling
Savepoint
compatibility Checkpoints
to savepoints
Connectors in Flink
Stream SQL
Windows
Large state
Maintenance
Fine grained
recovery
Side in-/outputs
Window DSL
Security
Mesos &
others
Dynamic Resource
Management
Authentication
Queryable StateApache Bahir connectors
Operations
Ecosystem
Application
Features
Broader
Audience
9. Flink 1.2 Improvements
5
Session Windows(Stream) SQL
Library
enhancements
Metric
System
Operations
Ecosystem
Application
Features
Metrics &
Visualization
Dynamic Scaling
Savepoint
compatibility Checkpoints
to savepoints
Connectors in Flink
Stream SQL
Windows
Large state
Maintenance
Fine grained
recovery
Side in-/outputs
Window DSL
Broader
Audience
Security
Mesos &
others
Dynamic Resource
Management
Authentication
Queryable StateApache Bahir connectors
10. Security / Authentication - Flink 1.2
6
Authorized data access
Secured clusters with Kerberos-based authentication
• Kafka, ZooKeeper, HDFS, YARN, HBase, …
Encrypted traffic between Flink Processes
• RPC, Data Exchange, Web UI, … - „SSL for all connections“
Largely contributed by
Prevent malicious users to hook into Flink jobs
12. Cluster Management - Flink 1.2
8Mesos integration contributed by
Standalone
Flink on Yarn
Flink on Mesos
13. Cluster Management - Beyond 1.2
9
Efforts to seamlessly interoperate with various
cluster managers.
Generalized abstraction (FLIP-6).
Driven by and
35. Many more improvements…
▪ Kafka 0.10 (with watermarks)
▪ Bucketing Sink: divides output into different file w.r.t. user
logic
15
36. Many more improvements…
▪ Kafka 0.10 (with watermarks)
▪ Bucketing Sink: divides output into different file w.r.t. user
logic
▪ Detached execution: first step in programatically controlled
job
15
37. Many more improvements…
▪ Kafka 0.10 (with watermarks)
▪ Bucketing Sink: divides output into different file w.r.t. user
logic
▪ Detached execution: first step in programatically controlled
job
▪ Async IO operator: non-blocking queries to external systems
15
38. Many more improvements…
▪ Kafka 0.10 (with watermarks)
▪ Bucketing Sink: divides output into different file w.r.t. user
logic
▪ Detached execution: first step in programatically controlled
job
▪ Async IO operator: non-blocking queries to external systems
▪ Improved scalability, robustness + bugfixes
15
46. Queryable State - Implementation
22
Query Client
State
Registry
window()
/
sum()
Job Manager Task Manager
ExecutionGraph
State Location Server
deploy
status
Query: /job/operation/state-name/key
State
Registry
Task Manager
(1) Get location of "key-partition"
for "operator" of" job"
(2) Look up
location
(3)
Respond location
(4) Query
state-name and key
local
state
register
window()
/
sum()
49. Queryable State Enablers
▪ Flink has state as a first class citizen
▪ State is fault tolerant (exactly once semantics)
23
50. Queryable State Enablers
▪ Flink has state as a first class citizen
▪ State is fault tolerant (exactly once semantics)
▪ State is partitioned (sharded) together with the
operators that create/update it
23
51. Queryable State Enablers
▪ Flink has state as a first class citizen
▪ State is fault tolerant (exactly once semantics)
▪ State is partitioned (sharded) together with the
operators that create/update it
▪ State is continuous (not mini batched)
23
52. Queryable State Enablers
▪ Flink has state as a first class citizen
▪ State is fault tolerant (exactly once semantics)
▪ State is partitioned (sharded) together with the
operators that create/update it
▪ State is continuous (not mini batched)
▪ State is scalable (e.g., embedded RocksDB state
backend)
23
61. Scaling Stateless Jobs
28
Scale Up Scale Down
Source
Mapper
Sink
• Scale up: Deploy new tasks
• Scale down: Cancel running tasks
62. Scaling Stateful Jobs
29
?
• Problem 1: Which state to assign to new task?
• Problem 2: Read + filter whole state?
63. Non-keyed vs Keyed State
30
• State bound to an operator + key
• E.g. Keyed UDF and window state
• „SELECT count(*) FROM t GROUP BY t.key“
• State bound only to operator
• E.g. Source state
KeyedNon-keyed
64. Non-keyed vs Keyed State
30
• State bound to an operator + key
• E.g. Keyed UDF and window state
• „SELECT count(*) FROM t GROUP BY t.key“
• State bound only to operator
• E.g. Source state
KeyedNon-keyed
65. Repartitioning Non-keyed state
31
#1 #2
#3 #4
#1 #2
#3 #4
Flink 1.1:
T snapshot()
void restore(T)
Flink 1.2:
List<T> snapshot()
void restore(List<T>)
Idea: break up state into finer granules that can be redistributed independently
66. Example: Kafka Source Flink 1.1
32
partitionId: 1, offset: 42
partitionId: 3, offset: 10
partitionId: 6, offset: 27
?
Operator state is black box. How to repartition?
67. Example: Kafka Source Flink 1.2
33
partitionId: 1, offset: 42
partitionId: 3, offset: 10
partitionId: 6, offset: 27
?
?
?
Return a list of sub-states which can be freely repartitioned.
72. Non-keyed vs Keyed State
36
• State bound to an operator + key
• E.g. Keyed UDF and window state
• „SELECT count(*) FROM t GROUP BY t.key“
• State bound only to operator
• E.g. Source state
KeyedNon-keyed
73. Repartitioning Keyed State
▪ Split key space into
key groups
▪ Every key falls into
exactly one key group
▪ Assign key groups to
tasks
37
Key space
Key group #1 Key group #2
Key group #3Key group #4
One key
74. Repartitioning Keyed State (ct’d)
▪ Rescaling changes
key group assignment
▪ Maximum parallelism
defined by #key
groups
38
75. Current State in Flink 1.2
▪ Manual rescaling
1. Take savepoint
2. Restart job with adjusted parallelism and
savepoint
39
76. Next Steps beyond Flink 1.2
▪ Rescaling individual operators w/o restart
▪ Refactor Flink deployment and process
model (previously discussed)
▪ On-the-fly Scaling
40
77. Autoscaling Policies
41
• Latency
• Throughput
• Resource utilization
• Kubernetes on GCE, EC2 and Mesos (marathon-
autoscale) already support auto-scaling