Apache Flink Overview presented by Jamie Grier in Flink Meetup at MaR Headquarters. https://community.mapr.com/docs/DOC-1503/edit?draftID=3848

1. What’s new in
Apache FlinkTM 1.0
Kostas Tzoumas
@kostas_tzoumas

2. Flink 1.0
• March 8, 2016
• First release in 1.x.y series
• Initiates backwards compatibility for selected APIs
• More than 64 contributors
• More than 450 JIRAs resolved
2

3. Flink 1.0: major features
• Out of core state
• Savepoints
• CEP library
• Improved monitoring & Kafka 0.9 support
3

4. Interface stability
4

5. Out of core state
5

6. Out of core state
• Alternative to in-memory state
• Powered by RocksDB instances in Flink TMs
• Enabled by using the RocksDBStateBackend
• State limited by disk space only
• State checkpoints save RocksDB databases in
reliable store
6

7. Savepoints
7

8. Production deployments
• Maintaining stateful applications in production
settings comes with its own challenges
• Failures, code upgrades, cluster maintenance, …
• Streaming jobs cannot be simply stopped and
restarted
8

9. Reminder: fault tolerance
• At least once, at most once, exactly once
• Flink guarantees exactly-once processing
• Flink guarantees end to end exactly-once with
selected sources and sinks
• e.g., Kafka —> Flink —> HDFS

10. How? Checkpoints
• Flink guarantees fault tolerance by regularly taking checkpoints
of the application state without ever stopping the execution
• At failure, input stream is rewinded to the logical time of the last
checkpoint
10

11. Introducing savepoints
• A savepoint is a Flink checkpoint that (1) is taken by
the user, (2) is accessible externally, and (3) never
expires
• Command line save & resume interface
• Save: flink savepoint <JobID>
• Resume: flink run -s
<path/to/savepoint> <jobJar>
11

12. Savepoints and versions
• A savepoint saves a version of a stateful application at a
well-defined time
• E.g.: take snapshots of one application at well-defined
times
12

13. “Like git for state”
• Branch off from savepoints creating a tree of
running application versions
13

14. Essential for production
deployments
• Application code upgrades
• Flink version upgrades
• Maintenance, migration, debugging
• What-if simulations
• A/B testing
• Time travel
14

15. Complex Event
Processing
15

16. FlinkCEP
• What is Complex Event Processing?
• A catch-all term
• In our context: easily detect patterns in streams
16

17. 17

18. Pattern API
18

19. 19

20. 20

21. Other features in 1.0
• Support for Kafka 0.9 API (and hence MapR
Streams)
• Monitoring console: job submission, checkpoint
statistics, detecting bottlenecks
• See
http://flink.apache.org/news/2016/03/08/release-
1.0.0.html
21

22. Closing
22

23. Summary
• Flink 1.0: Initiating backwards compatibility and
pushing the envelope even further for production
streaming deployments
23

24. What’s next
• SQL
• Dynamic scaling (+ savepoints)
• Hybrid in-memory/out-of-core state backend
• Query-able state
• Support for Apache Mesos
• More connectors and sinks (Kinesis, Cassandra, …)
24

25. Join the community
• Follow: @ApacheFlink, @dataArtisans
• Read: flink.apache.org/blog, data-artisans.com/blog
• Subscribe: (news | dev | user)@flink.apache.org