DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
Deployment Design Patterns - Deploying Machine Learning and Deep Learning Models into Production
1.
2. Confidential & Proprietary
Deployment Design Patterns
Deploying Machine Learning Models into Production
Dan Zaratsian, Cloud Solutions Engineer @ Google
https://github.com/zaratsian
October 2018
3. Confidential & Proprietary
Deployment Design Patterns
1. Train ML Model (using sklearn)
2. Deploy as Batch
3. Deploy as Web App (also using Spark)
4. Deploy as Web Service (Serverless)
5. Automated Model Build & Deploy (NLP, News Sources)
4. Confidential & Proprietary
Why is this Important?
● Too much time is lost before deploying.
● Deployments are not scalable.
● Deployments are not easily maintained & updated.
● Deployed models are not monitored.
● Too many models live (and die) on laptops.
● Deployment process is not understood.
5. Confidential & Proprietary
Dan Zaratsian
Cloud Solutions Engineer @ Google
University of Akron
B.S. Electrical Engineering
North Carolina State University
M.S. Advanced Analytics
6. Confidential & Proprietary
Data Scientist Production
Environment
End Users
(Customers)
Value
ML Model ML Model
Ideal Scenario (Simplified)...
7. Confidential & Proprietary
Data Scientist Production
Environment
End Users
(Customers)
No Value
ML Model
A Typical Scenario (What is commonly done)...
Lots of really great code in a Notebook,
but may not easily deployed.
8. Confidential & Proprietary
Why is this a challenge?
A good model may look something like this...
Model Score
Code
Date Quarter Down YardsToGo PlayType
2018-09-16 1 4 6 Run
2018-09-23 4 1 10 Pass
2018-10-07 4 3 13 Pass
2018-10-14 3 1 10 Run
Predicted Yards Gained
6.15 yards
-0.90 yards
3.95 yards
1.50 yards
Com h iv M
Pip e
(mo b e t / s o c e)
9. Confidential & Proprietary
Pre c in & Fe t e E g e n
Why is this a challenge?
It’s common to use dummy variables, standardize values, transform, etc...
Model Score
Code
Date Quarter Down YardsToGo PlayType
2018-09-16 1 4 6 Run
2018-09-23 4 1 10 Pass
2018-10-07 4 3 13 Pass
2018-10-14 3 1 10 Run
Predicted Yards Gained
6.15 yards
-0.90 yards
3.95 yards
1.50 yards
Month Quarter Down
YardsToG
o
PlayType_
Run
PlayType_
Run
9 1 4 6 1 0
9 4 1 10 0 1
10 4 3 13 0 1
10 3 1 10 1 0
In o p (or )
M ip e
10. Confidential & Proprietary
Why is this a challenge?
Or in many cases, the input is a sparse matrix, heavily feature engineered...
Date Quarter Down YardsToGo PlayType
2018-09-16 1 4 6 Run
2018-09-23 4 1 10 Pass
2018-10-07 4 3 13 Pass
2018-10-14 3 1 10 Run
Predicted Yards Gained
6.15 yards
-0.90 yards
3.95 yards
1.50 yards
Pre c in & Fe t e E g e n
Model Score
Code
In o p (or )
M ip e
11. Confidential & Proprietary
A few thoughts...
● Model deployment can mean different things across organizations.
● It’s complex and there’s a need for cross domain knowledge.
● Requirements vary by use case, industry, organization.
● There’s typically no one right way to deploy models.
12. Confidential & Proprietary
Data Scientist Production
Environment
End Users
(Customers)
Value
ML Model ML Model
Deployment Considerations
● What tool was used?
● What programming language?
● What are the dependencies?
● Is the model pipelined generalized?
● Can it easily be deployed across OS,
clouds, on-prem, etc?
● Is there a process for
promoting new models
into production?
● How often are models
updated or retrained?
● Where is this server(s) hosted?
● How do we scale?
● How do we secure our apps?
● How do we govern the pipeline?
How do end
users consume
our model?
13. Confidential & Proprietary
● How will your models be developed (code, drop-and-drag, hybrid)?
● How will your models be deployed?
○ As a batch process
○ Available via API
○ Real-time / Online Stream
○ Recoded (with model coefficients, logic, etc.)
● What scheduler is being used?
● Is your scheduler able to load and deploy your model pipeline?
● How often will the model be retrained?
● Do you have a process in place to monitor model performance?
● Using docker & kubernetes, serverless, ...?
Deployment Considerations (Continued)
14. Confidential & Proprietary
Deployment Design Patterns
1. Train ML Model (using sklearn)
2. Deploy as Batch
3. Deploy as Web App (also using Spark)
4. Deploy as Web Service (Serverless)
5. Automated Model Build & Deploy (NLP, News Sources)
15. Train ML Model
Deployment Design Patterns
1. Train ML Model (using sklearn)
2. Deploy as Batch
3. Deploy as Web App (also available using Spark)
4. Deploy as Web Service (Serverless)
5. Automated Model Build & Deploy (NLP, News Sources)
16. Confidential & Proprietary
ML Model Training Process (Simplified)
Data Prep &
Feature Eng.
Model
(model.joblib)
Model
Training
Model
Evaluation
ML Client Environment
Data
Scientist
Database
End Users
(Customers)
Training
Data
17. Batch Deployment
Deployment Design Patterns
1. Train ML Model (using sklearn)
2. Deploy as Batch
3. Deploy as Web App (also available using Spark)
4. Deploy as Web Service (Serverless)
5. Automated Model Build & Deploy (NLP, News Sources)
18. Confidential & Proprietary
Reference Architecture: Batch Scoring
Data Prep &
Feature Eng.
Model
(model.joblib)
Data
Scientist
Model
Training
Model
Evaluation
ML Client Environment
End Users
(Customers)
Training
DataDatabase
19. Confidential & Proprietary
Reference Architecture: Batch Scoring
Data Prep &
Feature Eng.
Model
(model.joblib)
Data
Scientist
Model
Training
Model
Evaluation
ML Client Environment
End Users
(Customers)
Model
(model.joblib)
Training
DataDatabase
Scored Records
Database Records
Deployed on Server, executed with Scheduler
20. Deploy as Web App
Deployment Design Patterns
1. Train ML Model (using sklearn)
2. Deploy as Batch
3. Deploy as Web App (also in Spark)
4. Deploy as Web Service (Serverless)
5. Automated Model Build & Deploy (NLP, News Sources)
21. Confidential & Proprietary
Reference Architecture: Deploy as Web App
Data Prep &
Feature Eng.
Model
(model.joblib)
Data
Scientist
Model
Training
Model
Evaluation
ML Client Environment
End Users
(Customers)
Training
DataDatabase
22. Confidential & Proprietary
Accept and
Process Request
POST
Reference Architecture: Deploy as Web App
Data Prep &
Feature Eng.
Model
(model.joblib)
Data
Scientist
Model
Training
Model
Evaluation
ML Client Environment
End Users
(Customers)
Model
(model.joblib)
Training
DataDatabase
Response
Database Records
Web Framework
(Simplified Architecture)
24. Deploy as Web Service (Serverless)
Deployment Design Patterns
1. Train ML Model (using sklearn)
2. Deploy as Batch
3. Deploy as Web App (also in Spark)
4. Deploy as Web Service (Serverless)
5. Automated Model Build & Deploy (NLP, News Sources)
25. Confidential & Proprietary
Reference Architecture: Deploy as Web Service (API)
Data Prep &
Feature Eng.
Model
(model.joblib)
Data
Scientist
Model
Training
Model
Evaluation
ML Client Environment
End Users
(Customers)
Training
DataDatabase
26. Confidential & Proprietary
Accept and
Process Request
API Call
Reference Architecture: Deploy as Web Service (API)
Data Prep &
Feature Eng.
Model
(model.joblib)
Data
Scientist
Model
Training
Model
Evaluation
ML Client Environment
End Users
(Customers)
Model
(model.joblib)
Training
DataDatabase
Response
Web Service (API)
(Simplified Architecture)
28. Automated Model Build & Deploy
Deployment Design Patterns
1. Train ML Model (using sklearn)
2. Deploy as Batch
3. Deploy as Web App (also in Spark)
4. Deploy as Web Service (Serverless)
5. Automated Model Build & Deploy (NLP, News)
29. Confidential & ProprietaryConfidential & Proprietary
UPDATEDEPLOYEVALUATETUNE ML MODEL
PARAMETERS
ML MODEL DESIGN
DATA
PREPROCESSING
Cloud AutoML
A technology that can automatically creates a Machine Learning Model
UPDATEDEPLOYEVALUATETUNE ML MODEL
PARAMETERS
ML MODEL
DESIGN
DATA
PREPROCESSING
30. Confidential & Proprietary
Why Cloud AutoML?
Your own
custom models
Simple
Limited ML
expertise needed
High quality
Confidential & Proprietary
32. Confidential & Proprietary
Deployment Design Patterns
1. Train ML Model (using sklearn)
2. Deploy as Batch
3. Deploy as Web App (also using Spark)
4. Deploy as Web Service (Serverless)
5. Automated Model Build & Deploy (NLP, News Sources)