Apache Flink @ Alibaba, by Haitao Wang, at Seattle Apache Flink Meetup (01/17/2018)

1. Flink at Alibaba
Haitao Wang

2. About me
• 2017.3 – present Alibaba Compute Platform
• 2012 – 2017.3 Microsoft China
• Spark as a Service, with a focus on streaming
• Skype for Business data
• 2000 – 2012 Microsoft US
• SQL Server engine
• Speech API & Speech Server
2

3. About Alibaba

4. About Alibaba Data
EBs Total PBs Everyday 100Ms Events/secTs Events/Day

5. Web Tier
DB Tier MQ
DataHub
Data Pipeline HBase Dashboard
Exactly-Once
Lots of Events
Sub second
latency
Why Streaming?
Highly Available

6. Challenges of Data Infra at Alibaba
Lots of JobsTons of Data
Exactly-OnceComplex Logic
Thousands of Machines Strict SLA
Low LatencyHigh Throughput

7. Introducing Alibaba Blink
+ = Blink
Apache Flink Alibaba’s Improvements Alibaba Blink

8. Blink numbers
+ = Blink
Apache Flink Alibaba’s Improvements Alibaba Blink
Unprecedented scale on 2017-11-11
472M events per second 10s of milliseconds latency Accurate results

9. Improvements in Blink Runtime
Async IO Increment CP Process & Deployment Metric

10. Declarative Optimizable Understandable
Why SQL?
Stable Unify

11. Dynamic Table
Stream Data
Blink
Batch Data
Dynamic
Table
Continuous
Query
Dynamic
Table
Stream Data
Batch Data
Stream Job
Batch Job

12. SQL Improvements
UDF/UDTF/UDAF Stream JOIN, etc. Retraction
Window AGG DML: INSERT etc. DDL

14. 2016
Scalability & Reliability
2017
Productivity

15. Use Case — Real-time A/B Test (Analytics)
Parser
Parser
Parser
Filter
Filter
Filter
Join UDF Agg
Impression
Click
Transaction
Druid
Online Logs

16. Use Case — Search Index Build & Update
HBase ClusterMysql Cluster
Mysql
IC2
Mysql
IC3
Mysql
IC1
Mysql
UIC2
Mysql
UIC3
Mysql
UIC1
HBase
IC
HBase
UIC
HBase
Result
Engine
search
Sync Join Export

17. Use Case — Online Machine Learning
Online Logs
HDFS
Feature
Feature
Compute
Online
Learning Model
Export Engine
online

18. Flink Advantages
• Event time / processing time
• Watermark, window, trigger
• Temporal join, retraction, …
Streaming semantics
• Managed state
• Exactly once
• Distributed checkpoints
• Queryable stable
Fault tolerant stateful processing at scale
• Process Functions, DataStream / DataSet, Table API / SQL
• Streaming, batch, event driven apps, CEP, ML, graph
Rich programming models and APIs

19. Blink Focuses
• Single job 10K+ parallelism, 10s TB state size
Scale
• Efficient resource utilization (FLIP-6)
• Runtime improvements
• Metrics & monitoring
Performance, costs and SLA
• SQL
• Platform – develop, debug, deploy, monitor, migration
• Connectors
Productivity

20. Blink Ecosystem in Alibaba
Cluster Resource Management
Search
Storage
StreamCompute Platform
Blink
Alibaba Apps Recommendation BI Lots more
DataStream API
Runtime Engine
Ads
DataSet API
SQL & Table API
Machine Learning Platform

21. Blink & Flink
YARN (Resource Management)
HDFS (Persistent Storage)
Blink Runtime
YARN App Master
Resource Manager
Job Manager
Task Manager
Tasks
Rocksdb State Backend
Web Monitor
Flink Client
Alibaba Data Lake
Submit Job
Launch AM Request TM Launch TM
Metrics
Apache Flink Alibaba Blink
Alibaba Monitor SystemMetric Reporter
Connectors
Read/Write
Checkpoint Incrementally
Debug
Task Scheduling
Checkpoint
Coordination

22. Examples of recent Blink runtime optimizations
• Credit-based network stack
• Dynamic load balancing
• Improved check-pointing for large scale jobs
• Some of these work has been contributed to Flink and will be released
in flink-1.5

23. Some New Flink Features
• Incremental Checkpoints
• Fine grained recovery
Flink 1.3
• Queryable states improvements
• Table API & SQL enhancements
Flink 1.4
• FLIP-6
• Network stack improvements
• State replication
• Eager State Declaration
• State evolution
Flink 1.5+

24. Evolution of Large State Handling & Recovery
24

25. G
H
C
D
Full Checkpoints
25
Checkpoint 1 Checkpoint 2 Checkpoint 3
I
E
A
B
C
D
A
B
C
D
A
F
C
D
E
@t1 @t2 @t3
A
F
C
D
E
G
H
C
D
I
E

26. G
H
C
D
Incremental Checkpoints
26
Checkpoint 1 Checkpoint 2 Checkpoint 3
I
E
A
B
C
D
A
B
C
D
A
F
C
D
E
E
F
G
H
I
@t1 @t2 @t3

27. Incremental Checkpoints
27
Checkpoint 1 Checkpoint 2 Checkpoint 3 Checkpoint 4
C1 C3C1 C1
Chunk
1
Chunk
2
Chunk
3
Chunk
4
Storage
C2 C4C3

28. Network Stack Improvements
• Removal of redundant copy operations
• Event driven network transfer
• Removal of artificial latency source
• Introduction of flow control
• Better control of back pressure
28

29. State Replication
• Decouple state from Tasks
• Faster recovery in case of Task failure
• Replicate state between Task Managers
• Faster failure recovery in case of machine failure
• High throughput queryable state
29

30. Thank You!
We are hiring…