Containerized deployments have taken the world by storm. Containers make your application portable across different machines and operating systems. They allow to scale applications in a matter of seconds. And they significantly simplify and speed up deployments which decreases development and operating costs. Consequently, more and more users want to deploy Apache Flink in containerized environments. In order to make Flink ready for container deployments, the Flink community has completely reworked Flink’s distributed architecture and added RESTful client-server communication. Moreover, the community added a container entry point with which it is a child’s play to build container images to get started. In this talk, we will take a closer look at the improvements to Flink which make it a first class citizen of the container world. We will demonstrate how easy it is to set up a Flink cluster on Kubernetes and discuss common pitfalls. We will conclude the talk with an outlook on future developments such as native Kubernetes support.

1. - GARY YAO
FLINK AS A LIBRARY
(AND STILL AS A FRAMEWORK)

2. © 2018 data Artisans2
ABOUT DATA ARTISANS
Original Creators of
Apache Flink®
RealTime Stream Processing Enterprise
Ready

3. © 2018 data Artisans3
FREE TRIAL DOWNLOAD
data-artisans.com/download

4. © 2018 data Artisans4
MOTIVATION

5. © 2018 data Artisans5
FLINK DEPLOYMENT OPTIONS & LIMITATIONS
BeforeApache Flink 1.5
• Shortcomings:
‒ Fixed number ofTaskManagers
‒ Difficult to support autoscaling
‒ Supporting new cluster managers difficult
‒ Akka for client—cluster communication
‒ Undisciplined about Job clusters
Session cluster Job cluster
Standalone ✓
YARN ✓ ✓
Mesos ✓
Kubernetes ✓

6. © 2018 data Artisans6
FLINK AS A LIBRARY
• Make deployments easier
• Application-oriented instead of cluster-oriented
• Just a start one or more Job jars, and automatically get distributed execution
• Remove distinction between different roles in the cluster (JM vs.TM)
• Participants of the cluster should determine roles on their own
‸

7. © 2018 data Artisans7
RESOURCE MANAGEMENT & AUTOSCALING
• Short-lived & high latency
• Resources only exist for the duration of
the job
• Different stages in a job may require
different resources
• Flink should be in control of
allocating/releasing resources
• Long-lived & low latency
• Dependent on external systems & exposed to
varying load
• External system should drive resource allocation
to meet SLO/SLAs
• Flink should react to available resources
Real-time
Applications
Batch Analytics
Continuous
processing

8. © 2018 data Artisans8
CONTAINERIZED DEPLOYMENT
Why Flink on containers?
• Self-contained runtime environment that includes application code, libraries,
dependencies, and configuration files
• Ease of operations by clean separation of concerns
• Enables dynamic resource allocation
• Fits the “Flink as a Library” model

9. © 2018 data Artisans9
CONTAINERIZED DEPLOYMENT (CONT.)
Brief introduction to Kubernetes
• De-facto industry standard for container orchestration
• Resource-oriented with declarative configuration
‒ You tell K8s the desired state, and a background process asynchronously makes it happen
‒ “3 replicas of this container should be kept running”
‒ “a load balancer should exist, listening on port 443, backed by containers with this label”
• Core resource types:
‒ Pod: a group of one or more containers running on a node
‒ Deployment: keep n pods running

10. © 2018 data Artisans10
REWORKING FLINK’S RUNTIME
FLIP-6 – Flink Deployment and Process Model (Flink 1.5 and up)
• Revamped distributed architecture
‒ Resource elasticity
‒ Support for different environments
‒ RESTfulAPI for client-cluster communication
• Building blocks:
ResourceManager Dispatcher
JobManager TaskManager
• Cluster manager specific
• Acquires/releases resources
• Touch point for job submissions
• Spawns JobManagers
• Single job only
• Deploys/monitors job execution
• Offers resources to ResourceManager
• Gets tasks from JobManagers

11. © 2018 data Artisans11
UPCOMING CHANGES

12. © 2018 data Artisans12
CONTAINER MODES
• Different levels of interaction with cluster manager
• Active Container Mode
‒ Flink is aware about the cluster manager that it is running on, and
interacts with it
‒ Already exists, e.g., FLIP-6YARN
• Reactive Container Mode
‒ Flink is oblivious to its environment
‒ Similar to standalone deployment
‒ Flink may react on new resources by scaling up job

13. © 2018 data Artisans13
ACTIVE K8S INTEGRATION
K8s deployment
controller
Client
TaskManager
JobManager
K8sResourceManager
ApplicationMaster
TaskManager
(3) Submit job
(1) SubmitAM deployment
(4) Start JM
(5) Request slots
(7) StartTM pod

14. © 2018 data Artisans14
REACTIVE CONTAINER MODE
• Relies on external system to start/release
TaskManagers, e.g.,
‒ Kubernetes Horizontal Pod Autoscaler
‒ GCPAutoscaling
‒ AWS Auto ScalingGroup
• Re-scale job if resources are added/removed
(take savepoint and resume job with new
parallelism automatically)
• By definition works with all cluster managers
Flink cluster
JM TM TM
ASG
Start new TM if
CPU% > threshold
Register
& offer
slots
Event rate over time

15. © 2018 data Artisans15
DOES ANY OF THIS WORK?
Demo: Autoscaling in Active & Reactive Container Mode

16. © 2018 data Artisans16
CURRENT STATE
• Recommendations on how to deploy job and session cluster on K8s
‒ Docker image build script
‒ K8s resource configs
• StandaloneJobClusterEntryPoint to start a (containerized) job cluster
• Work in progress/planned:
‒ Reactive Container Mode with automatically rescaling
• Design documents will be published soon
‒ Active K8s integration: prototype is available
• git clone -b nativeKubernetes git@github.com:tillrohrmann/flink.git

17. © 2018 data Artisans17
SUMMARY
• Thinking in terms of jobs & application code as an alternative to deploying Flink clusters
• Active K8s integration & Reactive Container Mode
• Existing code already enables job and session clusters on K8s
• Call to action:
‒ Umbrella tickets: FLINK-9953, FLINK-7087
‒ Follow dev@flink.apache.org to stay informed about latest state of development

18. THANK YOU!
@gjyao
@dataArtisans
@ApacheFlink
WE ARE HIRING
data-artisans.com/careers