In this presentation, drawing upon Thorogood’s experience with a customer’s global Data & Analytics division as their MLOps delivery partner, we share important learnings and takeaways from delivering productionized ML solutions and shaping MLOps best practices and organizational standards needed to be successful. We open by providing high-level context & answering key questions such as “What is MLOps exactly?” & “What are the benefits of establishing MLOps Standards?” The subsequent presentation focuses on our learnings & best practices. We start by discussing common challenges when refactoring experimentation use-cases & how to best get ahead of these issues in a global organization. We then outline an Engagement Model for MLOps addressing: People, Processes, and Tools. ‘Processes’ highlights how to manage the often siloed data science use case demand pipeline for MLOps & documentation to facilitate seamless integration with an MLOps framework. ‘People’ provides context around the appropriate team structures & roles to be involved in an MLOps initiative. ‘Tools’ addresses key requirements of tools used for MLOps, considering the match of services to use-cases.

1. Experimentation to
Industrialization:
Implementing MLOps
Deb Lee & Al McEwan
Thorogood Associates

2. About Us
Deb Lee
SENIOR CONSULTANT
MLOps Practice Lead
deb.lee@thorogood.com
Al McEwan
PRINCIPAL CONSULTANT
Solutions Architect, Databricks Champion, Global
Head of Capability Development
al.mcewan@thorogood.com
Independent, Specialist Data & AI Consultancy
US • UK • Singapore • Brazil • India
Databricks Partner Since 2018
www.thorogood.com
Data Science MLOps & DevOps
Data Engineering Data Visualization

3. Agenda
ØContext
ØCase Study
ØKey Learnings &
Takeaways
§ People
§ Processes
§ Tools

4. Companies are investing heavily in ML and AI

5. Thinking must shift to embrace operationalization
• Sandbox environments
• Ad Hoc, Exploratory
• Low Commitment
• Familiar tool for the data
scientist
• Done locally or in non-
integrated environments
EXPERIMENTAL
• Automated
• Integrated
• Reusable
• Scalable
• Understood and trusted
• Cost efficient
• Ongoing experiments
OPERATIONAL
• Sandbox environments
• Ad Hoc, Exploratory
• Low Commitment
• Familiar tool for the data
scientist
• Done locally or in non-
integrated environments
EXPERIMENTAL
• Automated
• Integrated
• Reusable
• Scalable
• Understood and trusted
• Cost efficient
• Ongoing experiments
OPERATIONAL
• Sandbox environments
• Ad Hoc, Exploratory
• Low Commitment
• Familiar tool for the data
scientist
• Done locally or in non-
integrated environments
EXPERIMENTAL
• Automated
• Integrated
• Reusable
• Scalable
• Understood and trusted
• Cost efficient
• Ongoing experiments
OPERATIONAL
MLOps

6. Key Benefits of MLOps
SCALABILITY
Ability to scale horizontally and vertically,
consumption efficiencies from running data
engineering and data science at-scale
MODEL EVALUATION
Maintain and monitor model quality using standardized &
consolidated custom KPIs and model evaluation metrics
FAST FEEDBACK LOOP
Respond to business opportunities and changes
quickly, incorporate enhancements to product on
regular basis
REUSABLE ASSETS
Track, monitor, and identify reusable assets
(registered models, datasets, pipelines) to
increase efficiency & cost savings
MODEL TRACEABILITY
Create traceability & wider auditability using enterprise
model registries, experiment tracking, and monitoring
operations for greater observability
AUTOMATED MODEL TRAINING
Decrease manual dependencies using pipelines
configured to kick off automated retraining based on
defined triggers
REPRODUCIBILITY
Save time & create governance for product teams
by using tools that enable reproducibility of
experiments and model training
VERSION SECURITY & COMPATABILITY
Maintain security by using licensed packages on
tested versions, keep OS versions of clusters up to
date, keep all libraries and packages up to date

7. Case Study

8. Establishing a Global MLOps Framework
Customer situation
In order to stay ahead, the customer recognized that a global coordinated
strategy and framework was needed to realize the benefits of MLOps
Investment in experimentation that has proven
valuable
Data science teams work in focused business
areas, following independent practices
Fortune Global 500
Consumer Goods
Company
• 190 countries
• 2.5 billion+ consumers
daily
• 400 brands

9. Establishing a Global MLOps Framework
Thorogood’s approach
Experimentation
ML models
operationalized
MLOps guidance,
recommendations &
artefacts, project-tested
Creation of reusable
Code & Pipeline
Accelerator templates

10. Establishing a Global MLOps Framework
Framework impact
REUSABILITY
As more products are onboarded, a central
function will improve reusability of existing assets
and help consolidate models and approaches
used across products.
TIME & COST SAVINGS
Reduce duplicative effort & apply responsible
cloud consumption principles to all projects,
receive cost efficiencies from consolidation of
operations.
SIMPLIFICATION
A centralized function will maintain
adherence to MLOps suggested standards to
simplify toolsets used and improve ways of
working for all teams.
CONTINUOUS IMPROVEMENT
The MLOps service will have dedicated teams
for ongoing operations and one-off activities
such as product enhancements &
industrialization efforts.
SCALABILITY
Enable data science projects to scale up
more quickly, rapidly realize a vision to
unlock business value using data science
in all areas of the organization.
RELIABILITY
Build greater trust and confidence from business
users and data science teams by allowing them
to realize the value of MLOps delivered using a
consistent and high-quality methodology
Customer’s
Global MLOps
Service

11. Key Takeaways & Learnings

12. Key Takeaways & Learnings
People Processes Tools
Data

13. People
Real-world ML Systems
Reference: “Hidden Technical Debt in Machine Learning Systems” by D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips 2015
Configuration
Data Collection
Feature Extraction
ML
Code
Data Verification Machine
Resource
Management
Analysis Tools
Process Management Tools
Serving
Infrastructure
Monitoring

14. People
MLOps Requires Data Scientists who understand both Scale and Reproducibility
ML Code: could be relatively small, but key to success
Data Scientist skillset specialization
Training in making code scalable, efficient and reproducible

15. People
Blend of capabilities and skills needed depends on the engagement
Machine Learning Engineering
Data Science
Data Engineering
Data Visualization
Solution Architect
Program Management
Scenario 1
Operationalization of a
use case requiring:
• Real-time model
serving capabilities
• Web application
interface and backend
• Creation of data
engineering and data
science pipelines
• Scripted management
and versioning of
compute, datastore,
datasets, pipelines
Scenario 2
Continuous improvements
to baseline monitoring
operations requiring:
• Create automatically
refreshed monitoring
dashboards
• Enhance tracking of
and reporting on drift
and other scoring
metrics alongside
experiment tracking
• Design for various
target audiences: data
scientists, ML support
engineers, business
users

16. Processes
Key Takeaways & Learnings – Artefacts Created
There are a number of moving parts and handshakes needed for a centralized MLOps service to function and teams to be in sync.
Without a defined framework and process, it’s hard to be successful.
QUESTIONNAIRE
Used to qualify use
cases & projects in the
pipeline for
onboarding to MLOps
service
ML TEST SCORE
Measures the overall
readiness of the ML
system for production
DECISION TREE
For anyone embarking on a
data science project, guide on
tools to use considering
training volumes, libraries,
serving method,
parallelization, retraining
frequency
PLAYBOOK
Guidelines for
experimentation and
operationalization to
streamline the MLOps
process
REPRODUCIBILITY
CHECKLIST
Requires code versioning,
data versioning, model
versioning in model
registry, cluster
configuration, environment
specification
Reference: “The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction” by E.Breck et al. 2017

17. Tools
Decision trees helping to guide tool selection at critical junctures
How many models are being
built?
A large model spanning the
entire business
One model per dimension (i.e.
per product)
We recommend use of Spark’s
MLLib if model is trained on a
big dataset (>0.5GB)
We recommended use of
Spark’s MLLib if cross-
validation scenarios exist
Non-Spark options can be
considered for smaller training
datasets
Non-Spark options can be
considered for this scenario
Training & Evaluation
Orchestration
Deployment
Tracking
Experimentation Initial Industrialization
Model Monitoring &
Enhancements
Considerations:
Decision Points:
…
…
…
…
…
…
…
…

18. Tools
Databricks is Optimally Positioned to Support MLOps
Databricks Spark: Optimized for
large training data volumes per
model
Best-in-class and widely used for
data science experiments
Multi-Cloud ready:
available on Azure, AWS, and GCP
Unifies requisite data engineering &
data science capabilities with in-built
functions
MLFlow provides a powerful platform
to manage the ML lifecycle
Integrated with serving and reporting
technologies

19. How to get started
Ø Assess your current state
Ø Define your target state
Ø Refine your approach to People, Tools and Processes
Ø Educate yourself on the ‘art of the possible’
• Check out our MLOps Resource Hub for useful content at www.thorogood.com
• Most importantly, please reach out to us with any questions or feedback on this topic
CONTACT US
Deb Lee
deb.lee@thorogood.com
Al McEwan
al.mcewan@thorogood.com

20. Feedback
Your feedback is important to us.
Don’t forget to rate and review the sessions.