Cloud Native Night July 2019, Munich: Talk by Emil A. Siemes (@mesosphere, Principal Solution Engineer at Mesosphere) === Please download slides if blurred! === Abstract: Tired of managing infrastructure instead of creating exiting ml models? Learn what DC/OS can do for the data scientist. Join us next time: https://www.meetup.com/Cloud-Native-muc/events

1. Emil A. Siemes
Principal Solution Engineer EMEA

2. Provide public cloud-like services
from an open partner ecosystem,
delivered on any infrastructure.

3. Transport
Message Queue
Store
Distributed Database
Analyze
AI / ML + Real-time
Serve
Container
Orchestration
Mesosphere Accelerates Innovation for New Digital Services
On Hybrid Cloud Infrastructures
Infrastructure
Datacenter &
Edge Clouds
AWS EC2 Microsoft Azure Google Cloud
• “As-a-service” Automation
IT adopts any platform technology quickly,
securely & efficiently
• Unified Multi-Cloud Operations
Consistent & resilient services across
cloud, datacenter & edge
3

4. PHYSICAL INFRASTRUCTURE
MICROSERVICES, CONTAINERS, &DEV TOOLS
VIRTUAL MACHINES PUBLIC CLOUDS
DATA SERVICES, MACHINE LEARNING, &AI
Security &
Compliance
Application-Aware
Automation
Multitenancy
Hybrid Cloud
Management
100+
MORE
DatacenterEdge
Datacenter and Cloud as a Single Computing Resource
Powered by Apache Mesos
20+
MORE
Unified multi-cloud operations
Securely manage cloud, datacenter, and edge
infrastructures from a single control plane
4
Mesosphere DC/OS Delivers “As-a-Service” Automation
to Any Application Technology on Any Infrastructure
Intelligent resource pooling
Optimize workload density for highest utilization with
resource guarantees
3
Broad workload coverage
Run today & tomorrow’s applications including traditional
J2EE, containers, analytics & ML
1
Application-aware automation
Automate workload-specific operating procedures to “as-a-
Service” anything from Kubernetes to data services
2

5. What you want to be doing
9
Data
(clean)
Write awesome ML code
Train
(once)
Deploy
(not me)

6. 10
Sculley, D., Holt, G., Golovin, D. et al. Hidden Technical Debt in Machine Learning Systems
What you’re actually doing

7. Open Source Pipeline
Operationalizing a Machine
Learning model can be super hard.
It is a stage where most enterprise
Machine Learning projects fail. I
cannot tell you how many
companies I've talked to, who have
said their innovation teams had
devised these cool ML projects, but
they were struggling getting the ML
models into production. In this set
of courses, we will talk about how
to train, deploy, and predict with ML
models in a way that their
production ready. And finally, we
delve back into Machine Learning
theory.
Valliappa Lakshmanan.
Tech Lead for Big Data and Machine
Learning Professional Services on Google
Cloud Platform.

8. 1. Model Engineering 2. Model Training 3. Monitoring 4. Debugging 5. Model Serving

9. 1. Data Preparation using
Spark
7. Streaming of requests
...
Public Cloud Pipeline
Model Engineering 2. Model Training 3. Monitoring 4. Debugging 5. Model Serving

10. 1. Data Preparation using
Spark
7. Kafka stream of
requests
DIY Open Source Pipeline
1. Model
Engineering
2. Model Training 3. Monitoring 4. Debugging 5. Model Serving

11. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

12. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

13. © 2018 Mesosphere, Inc. All Rights Reserved. 21
Challenges
● Training/Dev/Test + New Data
● Large amounts
● Quality
● Availability (for cluster)
● Velocity
● Streaming
Solutions
GFS
Input Data Management
Input:
Lots of Labeled
Data
Apache Kafka
Apache Cassandra

14. © 2018 Mesosphere, Inc. All Rights Reserved. 22
Challenges
● Data is typically not ready to be
consumed by ML job*
● Data Cleaning
● Missing/incorrect labels
● Data Preparation
● Same Format
● Same Distribution
Solutions
Data Preparation
* Demo datasets are a fortunate exception :)

15. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

16. © 2018 Mesosphere, Inc. All Rights Reserved. 24
Challenges
● Different Users/Use cases
● Data Analyst/Exploring
● Production Workloads
● Highly Optimized
● How to spawn Environments?
Solutions
Users

17. © 2018 Mesosphere, Inc. All Rights Reserved. 25
Challenges
● Different Users/Use cases
● Data Analyst/Exploring
● Production Workloads
● Highly Optimized
● How to spawn Environments?
Solutions
Users

18. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

19. © 2018 Mesosphere, Inc. All Rights Reserved.
TensorFlow...
27

20. © 2018 Mesosphere, Inc. All Rights Reserved.
Resource Isolation and Allocation
30

21. © 2017 Mesosphere, Inc. All Rights Reserved. 31
Datacenter
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Mesos/ DC/OS
automated schedulers, workload multiplexing onto the
same machines
Tensorflow
Jenkins
Kafka
Spark
Tensorflow

22. © 2018 Mesosphere, Inc. All Rights Reserved.
Two-level Scheduling
1. Agents advertise resources to Master
2. Master offers resources to Framework
3. Framework rejects / uses resources
4. Agent reports task status to Master
34
MESOS ARCHITECTURE
Mesos
Master
Mesos
Master
Mesos
Master
Mesos AgentMesos Agent Service
Cassandra
Executor
Cassandra
Task
Flink
Scheduler
Spark
Executor
Spark
Task
Mesos AgentMesos Agent Service
Docker
Executor
Docker
Task
CDB
Executor
Spark
Task
Spark
Scheduler
Kafka
Scheduler

23. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

24. © 2018 Mesosphere, Inc. All Rights Reserved. 38
Challenges
● Many Models
● Different Hyperparameter
● Different Models
● New Training Data
● ...
Solutions
● Persistent Storage + Metadata
Model Management
GFS

25. Data Science Pipeline on DC/OS
Continuous Integration
Monitoring & Operations
Distributed Data
Storage and
Streaming
Data Preparation
and Analysis
Storage of trained
Models and
Metadata
Use trained Model
for Inference
Distributed
Training using
Machine Learning
Frameworks
Data & Streaming
Model
Engineering
Model
Management
Model Serving
Model
Training
Management
Tensorboard

26. © 2018 Mesosphere, Inc. All Rights Reserved. 40
Challenges
● How to Deploy Models?
● Zero Downtime
● Canary
Solutions
● Seldon-Core
● TensorFlow Serving
Model Serving

27. © 2018 Mesosphere, Inc. All Rights Reserved.
● DC/OS TensorFlow Package (currently closed source)
○ https://github.com/mesosphere/dcos-tensorflow
● DC/OS TensorFlow Tools
○ https://github.com/dcos-labs/dcos-tensorflow-tools/
● Tutorial for deploying TensorFlow on DC/OS
○ https://github.com/dcos/examples/tree/master/tensorflow
● Contact:
○ https://groups.google.com/a/mesosphere.io/forum/#!forum/tensorflow-
dcos
○ Slack: chat.dcos.io #tensorflow
Questions and Links
43