The overall evolution towards microservices has caused a lot of IT leaders to radically rethink architectures and platforms. One can hardly keep up with the rapid onslaught on new distributed technologies. The same people who just asked yesterday "how can we deploy Docker containers?", are now asking "how can we operate Kubernetes-as-a-Service on-premise?", and are about to start asking "how can we operate the open source frameworks of our choice, such as Spark, TensorFlow, HDFS, and more, as a service across hybrid clouds?”. This session will discuss: Challenges of orchestrating and operating

1. Hybrid Cloud
(Kubernetes, Spark, HDFS)-as-a-Service
@joerg_schad
Berlin | November 20 - 21, 2018

2. Jörg Schad
Technical Lead Data
Science
● Core Mesos
developer at
Mesosphere
● Twitter:
@joerg_schad

3. Hybrid Cloud
(Kubernetes, Spark, HDFS, …)-as-a-Service
@joerg_schad @mesosphere

4. Yesterday: Individual Containers
4

5. Present: On-Prem Container Orchestration
5

6. So you’ve decided you want to run Kubernetes ...
6

7. ETCD 01
MASTER 01
etcd
LoadBalancer
WORKER 01
API Server
Scheduler
Controller
Manager
Kubectl
Kubelet Kube Proxy
Container Engine (Docker,
CRI-O)
Pod Pod Pod
HTTPS
HTTPS
HTTPS
HTTPS

8. Deploying Kubernetes
1. Prerequisites
2. Installing the Client Tools
3. Provisioning Compute Resources
4. Provisioning the CA and Generating TLS Certificates
5. Generating Kubernetes Configuration Files for Authentication
6. Generating the Data Encryption Config and Key
7. Bootstrapping the etcd Cluster… 3x for HA
10. Bootstrapping the Kubernetes Control Plane… 3x for HA
13. Bootstrapping the Kubernetes Worker Nodes
14. Configuring kubectl for Remote Access
15. Provisioning Pod Network Routes
16. Deploying the DNS Cluster Add-on… Deploying other Add-ons
20. Smoke Test
21. Cleaning Up
Running on your own

9. © Gerard Julien/AFP
3 AM

10. Kubernetes Lifecycle Management
10
Deploy
Scale
Upgrade
Recover
...
1.9.3
1.9.4
1.9.6
1.9.7
1.10.3
1.10.4
...

11. Manual
- Exceptionally difficult
- DIY PRs dormant for
years
- Multiple steps
- Zero day-two
DIY Configuration
Management
- Easy “hello world”
- No commercial support
- DIY security
- DIY PRs dormant for
years
- Lacks auxiliary
components
Vendor Configuration
Management
- Easy provisioning of
clusters
- Hard coded
infrastructure
- Can include auxiliary
components
Cloud Provider
- Easy provisioning
multi-clusters
- Standard centralized
management
- Includes auxiliary
components
KOPS
Kubespray
Kubeadm
Public Anywhere
(Bare Metal, Cloud,
Virtualization)
Passive Intelligent
Google Container
Engine (GKE)
How to manage Kubernetes

12. Future: Hybrid Cloud Service Orchestration
12
Kafka Cassandra Spark HDFSCassandra
On-Prem AWS

13. Why Services?
13
Microservice
1. Microservice records
running activity

14. Why Services?
14
Microservice Display
1. Microservice records
running activity
2. Data written to
Kafka topic
3. Data analyzed Flink 4. Results stored in
Cassandra
7. Reminder displayed
using Microservice

15. Why Services?
Source Code
Control
Build and Test Release
Deploy, Monitor
and Log
Automatically trigger CI/CD
pipeline based on code
check-in.
Start automated build and test,
including functional, security
and performance tests.
Update artifact repository with
latest successful code artifacts
and pull newest images
Deploy applications to container
orchestration and watch with
monitoring and logging
CONTINUOUS INTEGRATION (CI) CONTINUOUS DEPLOYMENT (CD)
Marathon

16. Why Services?
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management

17. © Gerard Julien/AFP
Service Orchestration
Cassandra

18. 18
Distributed
Systems
Are
hard...

19. Container Orchestration Service Orchestration
Container: Deploy/Remove
E.g., ReplicaSet Controller
19
Replicas = 4
DESIRED
Replicas = 0
CURRENT
#Namenodes = 2
DESIRED
#Namenodes = 0
CURRENT
HDFS Namenode
1. Recover State from
Journal
2. Launch Namenode 1
3. Launch Namenode 2

20. State Machines

21. Challenge 1: Service Lifecycle Management
21
Deploy
Scale
Configure
Recover
...
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Kubernetes
Jenkins
Kafka
Kubernetes
TensorFlow

22. Deploying Kubernetes
1. Prerequisites
2. Installing the Client Tools
3. Provisioning Compute Resources
4. Provisioning the CA and Generating TLS Certificates
5. Generating Kubernetes Configuration Files for Authentication
6. Generating the Data Encryption Config and Key
7. Bootstrapping the etcd Cluster… 3x for HA
10. Bootstrapping the Kubernetes Control Plane… 3x for HA
13. Bootstrapping the Kubernetes Worker Nodes
14. Configuring kubectl for Remote Access
15. Provisioning Pod Network Routes
16. Deploying the DNS Cluster Add-on… Deploying other Add-ons
20. Smoke Test
21. Cleaning Up
Running on your own
$ dcos package install
kubernetes
on

23. Encrypted Communication
1. Generate certificates and keys using DC/OS PKI
2. Store certificates in DC/OS Secret Store
3. Distribute certificates and keys to data services
4. Apply security configuration
5. Perform rolling restart of the service
Each data service has its own distinct security procedure, which sometimes change per version
Kafka Cassandra Elastic Spark HDFS

24. Complete Lifecycle
$ dcos hdfs plan list
[
"deploy",
"recovery",
"replace",
"update"
]
dcos hdfs plan show deploy
deploy (serial strategy) (IN_PROGRESS)
├─ journal (parallel strategy) (IN_PROGRESS)
│ ├─ journal-0:[node] (COMPLETE)
│ ├─ journal-1:[node] (COMPLETE)
│ └─ journal-2:[node] (PREPARED)
├─ name (serial strategy) (PENDING)
│ ├─ name-0:[format] (PENDING)
│ ├─ name-0:[node] (PENDING)
│ ├─ name-1:[bootstrap] (PENDING)
│ └─ name-1:[node] (PENDING)
├─ zkfc (serial strategy) (PENDING)
│ ├─ name-0:[zkfc-format] (PENDING)
│ ├─ name-0:[zkfc] (PENDING)
│ └─ name-1:[zkfc] (PENDING)
└─ data (parallel strategy) (PENDING)
├─ data-0:[node] (PENDING)
├─ data-1:[node] (PENDING)
└─ data-2:[node] (PENDING)

25. 25

26. Challenge 2: Cluster Management
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Kubernetes
Jenkins
Kafka
Kubernetes
TensorFlow

27. Challenge 2: Cluster Management
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Mesos/ DC/OS
automated schedulers, workload multiplexing onto the
same machines
Kubernetes
Jenkins
Kafka
Kubernetes
TensorFlow

28. © 2018 Mesosphere, Inc. All Rights Reserved.
Powered by Apache Mesos

29. Stateful Services
Kelsey Hightower
Kubernetes Thought Leader
Ranked #1 K8s Influencer
Staff Developer Advocate
PM & Chief Advocate
Top
Kubernetes
Advocate

30. CRDs

31. CRDs
https://book.kubebuilder.io/

32. Challenge 3: Multi-tenancy
32
Powered by Apache Mesos

33. Multiple Kubernetes Cluster
33
Powered by Apache Mesos
Line of Business 1 Line of Business 2 Developer Cluster Shared Production
Skaffold

34. Multiple Kubernetes Cluster
34
Powered by Apache Mesos
Line of Business 1 Line of Business 2 Developer Cluster Shared Production
Skaffold

35. Resource Quota
35
• Share resources between
multiple services
• No static partitioning
• One role per job/entity
• Use quota per role
• Min and Max resource
allocation

36. Challenge 4: Hybrid Cloud
Powered by Apache Mesos

37. Why Hybrid
37

38. © 2018 Mesosphere, Inc. All Rights Reserved. 38
THANK YOU!
ANY
QUESTIONS?
@mesosphere
users@dcos.io
/groups/8295652
/dcos
/dcos/examples
/dcos/demos
chat.dcos.io

39. CONFIDENTIAL
Make it insanely easy
to build and scale
world-changing technology

40. Questions?
@joerg_schad @dcos
@dcos
users@dcos.io
/groups/8295652
/dcos
/dcos/examples
/dcos/demos
chat.dcos.io