Kubernetes is great for deploying stateless containers, but what about the big data ecosystem? Episode 3 of our Kubernetes series covers how DC/OS enables you to connect your Kubernetes-based applications to co-located big data services. Slides cover: 1. Why persistence is challenging in distributed architectures How DC/OS helps you take advantage of the services available in the big data ecosystem 2. How to connect Kubernetes to your data services through networking 3. How Apache Flink and Apache Spark work with Kubernetes to enable real-time data processing on DC/OS

1. Kubernetes and Big Data
Services
@joerg_schad @gaunetes @dcos

2. Chris Gaun
PMM at Mesosphere /
Kubernetes Expert /
CNCF Ambasador
● Previous to that
Gartner analyst
covering public IaaS
● Kubernetes
community for 3 years

3. © 2018 Mesosphere, Inc. All Rights Reserved.
Mesosphere DC/OS at KubeCon EU
● Mesosphere - Platinum
Sponsor
● Many presentation:
container storage, ML,
HDFS
● Demoing smart city
application
3

4. Jörg Schad
Technical Community
Lead / Developer
● Core Mesos
developer at
Mesosphere
● Passions are deep
learning, distributed
data systems, and
data analytics

5. Bootcamp: Building Kubernetes-as-a-Service at
Scale, Anywhere
● Episode 1: Building Kubernetes-as-a-Service
at Scale
● Episode 2: Deploying Kubernetes at Scale
with DC/OS
● Episode 3: Kubernetes and Big Data
Services
● Episode 4: Operating Kubernetes at Scale
with DC/OS
● End-to-end components
and best practices
● Automated management
of Kubernetes
● Connecting Kubernetes
to Big Data services
● Delivering an entire
Kubernetes solution

6. 6
Star / Clone Github
1. Go to Kubernetes DC/OS
quickstart
2. Search “DC/OS Kubernetes
Quickstart Github” or
https://github.com/mesospher
e/dcos-kubernetes-quickstart
3. Live demo
https://github.com/dcos/demos
/tree/master/flink-k8s/1.11

7. 7
Sign Up For Slack
1. Slack URL: https://chat.dcos.io/
2. Join #kubernetes channel
3. OSS support / feedback

8. © 2017 Mesosphere, Inc. All Rights Reserved. 8
MapReduce is
crunching Data
Ancient
Times...

9. © 2016 Mesosphere, Inc. All Rights Reserved. 9
But then business
demanded
FAST DATA
We need to turn faster!
Today...

10. © 2016 Mesosphere, Inc. All Rights Reserved. 10
Fast Data
Batch Event ProcessingMicro-Batch
Days Hours Minutes Seconds Microseconds
Solves problems using predictive and prescriptive analyticsReports what has happened using descriptive analytics
Predictive User InterfaceReal-time Pricing and Routing Real-time AdvertisingBilling, Chargeback Product recommendations

11. © 2016 Mesosphere, Inc. All Rights Reserved. 11
The SMACK Stack
EVENTS
Ubiquitous data streams
from connected devices
INGEST
Apache Kafka
STORE
Apache Spark
ANALYZE
Apache Cassandra
ACT
Akka
Ingest millions of events
per second
Distributed & highly
scalable database
Real-time and batch
process data
Visualize data and build
data driven applications
Apache Mesos/ DC/OS
Sensors
Devices
Clients

13. © 2016 Mesosphere, Inc. All Rights Reserved. 13
The SMACK Stack
EVENTS
Ubiquitous data streams
from connected devices
INGEST
Apache Kafka
STORE
Apache Flink
ANALYZE
Apache Cassandra
ACT
Akka
Ingest millions of events
per second
Distributed & highly
scalable database
Real-time and batch
process data
Visualize data and build
data driven applications
Apache Mesos/ DC/OS
Sensors
Devices
Clients

14. © 2017 Mesosphere, Inc. All Rights Reserved. 14
Challenges

15. © 2018 Mesosphere, Inc. All Rights Reserved. 15
Datacenter
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Kubernetes
Jenkins
Kafka
Spark
Cassandra

16. 3 AM
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Kubernetes
Jenkins
Kafka
Spark
Cassandra

17. DC1
DC2
DC3
3 AM

18. © 2017 Mesosphere, Inc. All Rights Reserved. 18

19. © 2018 Mesosphere, Inc. All Rights Reserved. 19
Datacenter
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Mesos/ DC/OS
automated schedulers, workload multiplexing onto the
same machines
Kubernetes
Jenkins
Kafka
Spark
Cassandra

20. • Brings “as-a-Service”
automation to any application
technology on any
infrastructure
• Organizations Run All Types of
Container Management as-a-
Service Using Mesos:
"(Netflix) launches up to 500,000
containers and 200,000
clusters/day"
-Netflix OSS, on using Titus container
management ontop Mesos

21. © 2018 Mesosphere, Inc. All Rights Reserved. 21
DC/OS
PHYSICAL INFRASTRUCTURE
MICROSERVICES, CONTAINERS, & DEV TOOLS
VIRTUAL MACHINES PUBLIC CLOUDS
DATA SERVICES, MACHINE LEARNING, & AI
Security &
Compliance
Application-Aware
Automation
Multitenancy
Hybrid Cloud
Management
100+
MORE
DatacenterEdge
Datacenter and Cloud as a Single Computing Resource
Powered by Apache Mesos
20+
MORE

22. © 2018 Mesosphere, Inc. All Rights Reserved. 22

23. © 2018 Mesosphere, Inc. All Rights Reserved.
Two-level Scheduling
1. Agents advertise resources to Master
2. Master offers resources to Framework
3. Framework rejects / uses resources
4. Agent reports task status to Master
23
MESOS ARCHITECTURE
Mesos
Master
Mesos
Master
Mesos
Master
Mesos AgentMesos Agent Service
Cassandra
Executor
Cassandra
Task
Kubernetes
Scheduler
Spark
Executor
Spark
Task
Mesos AgentMesos Agent Service
Docker
Executor
Docker
Task
K8s Executor
Kubelet
Task
Marathon
Scheduler
Kafka
Scheduler

24. © 2018 Mesosphere, Inc. All Rights Reserved.
Distributed Systems are ...
HDFS Scheduler

25. © 2018 Mesosphere, Inc. All Rights Reserved.
Plans
dcos hdfs --name=hdfs plan status deploy
deploy (serial strategy) (COMPLETE)
├─ journal (serial strategy) (COMPLETE)
│ ├─ journal-0:[node] (COMPLETE)
│ ├─ journal-1:[node] (COMPLETE)
│ └─ journal-2:[node] (COMPLETE)
├─ name (serial strategy) (COMPLETE)
│ ├─ name-0:[node, zkfc] (COMPLETE)
│ └─ name-1:[node, zkfc] (COMPLETE)
└─ data (serial strategy) (COMPLETE)
├─ data-0:[node] (COMPLETE)
├─ data-1:[node] (COMPLETE)
└─ data-2:[node] (COMPLETE)

26. Server Server Server Server Server
Server Server Server Server Server
as-a-Service
Installation
KubernetesOne-Click
20+
MORE
Cloud Native
Services

27. ServerServer Server
as-a-Service
Automated Self Healing
Server Server Server Server Server
Server Server Server
KubernetesZero
Touch
20+
MORE
Cloud Native
Services

28. © 2018 Mesosphere, Inc. All Rights Reserved. 28
Why {Spark, HDFS, ..} on K8s today?
Kelsey Hightower
Kubernetes Thought
Leader
Ranked #1 K8s Influencer
Staff Developer
Advocate
PM & Chief Advocate
Today Big Data on K8s is more DIY
Top
Kubernetes
Advocate

29. © 2018 Mesosphere, Inc. All Rights Reserved. 29
SMACK Stack
Generator Display
1. Financial data created
by generator
2. Written to
Kafka topics
3. Kafka Topics
consumed by Spark or
Flink
4. Results written back into Kafka
stream (another topic)
7. Results displayed

30. 30
Star / Clone Github
1. Go to Kubernetes DC/OS
quickstart
2. Search “DC/OS Kubernetes
Quickstart Github” or
https://github.com/mesospher
e/dcos-kubernetes-quickstart
3. Live demo
https://github.com/dcos/demos
/tree/master/flink-k8s/1.11

31. © 2018 Mesosphere, Inc. All Rights Reserved. 31
SMACK Stack
Generator Display
1. Financial data created
by generator
2. Written to
Kafka topics
4. Results written back into Kafka
stream (another topic)
5. Results displayed
3. Kafka Topics
consumed and analyzed
by Flink
Kubernetes Cluster
(running on top of DC/OS)

32. Download Now
https://mesosphere.com/resource/category/ebook/

33. © 2018 Mesosphere, Inc. All Rights Reserved. 33
THANK YOU!
ANY
QUESTION
S?
@dcos
users@dcos.io
/groups/8295652
/dcos
/dcos/examples
/dcos/demos
chat.dcos.io
https://github.com/mesosphere/dcos-kubernetes-quickstart
https://mesosphere.com/blog/another-kubernetes-service/