SlideShare ist ein Scribd-Unternehmen logo
1 von 23
Downloaden Sie, um offline zu lesen
Engineers guide for
shepherding models
in to production
Marko Dimitrijević
Staff software engineer at vroom
marko.dimitrijevic@vast.com
m_a_r_e.91@hotmail.com
What is an example of a
data science model?
● Takes in data
● Learns from data
● Gets updated
● Produces results
● Provides value
What does it mean to
run a model in
production
environment
And what does it mean to run a model in real time
● Returns a result every time (or a
meaningful error)
● Inputs and outputs are well
defined and explained
● Performant and reasonably
optimized for the task
● Results can be depended on and
are verified to be correct
● System is able to handle errors
and edge cases
Two main topics we will cover
Building model pipeline
● Defining inputs and outputs
● Defining a call place
● Collecting inputs and
delivering outputs
● Handling edge cases
Hosting the model
● Defining resource needs
● Optimizing model code
● Integrating with platform
code
● Testing and iterating
What does the
engineering team do
We also rename stuff to follow standards!
● We take care of monitoring and
scaling to match the load
● We take care of the model
hosting and plumbing for
providing inputs and outputs
● We also advise on what is
doable and not and help find the
optimal solution
● We help rework models to run
faster, better or to follow good
practices
● We know to ask the right
question and prevent future
problems
How model pipeline
is built
Assuming research part is done and algorithm is
picked
● Produce the model
● Identify where we can get the
data for training
● Find a way to get all the inputs
the model needs
● Find the right location to integrate
the model and call it
● Deliver model outputs to the
right place
When model trains on processed data
Not a good way to do it!
Common issues in building model pipelines
Most common issues are data related
● Model is using data upstream of the place from where it will be called.
This can lead to inputs that do not take raw data but some processed/aggregated version of it that is not
available at model call place.
● Data gets renamed many times on its way from source to the model training
Sometimes people call same things different names and vice versa, this can lead to confusion and wrong
inputs being used. If i want a location is that location of the customer or vehicle location or address of
reconditioning center handling the vehicle?
● Incomplete data is filtered out in training but can it be in real world
It is easy to ignore incomplete data when there is a lot of leftover data to work with, but in the real world pieces
of information are often missing and sometimes a prediction is better than no prediction. Coverage can be very
low for models that are strict about their inputs.
Common issues in building model pipelines
Potential timing issues
● If model is called in real time and user is waiting for a response it should be <1s
Sometimes models are not built to operate fast on a single input but are optimized for batch processing. Also
we often need to spend time querying different services to prepare model inputs and that time adds up to the
total response time to the user.
● Once model is done and needs to be integrated the process takes a long time
Inserting a model in a production flow can take time and if we need to build a pipeline that gets multiple
different and “exotic” inputs that can include many teams and be time consuming
● It is hard to identify the right moment in time to trigger the model
We often want to predict something as early in the process as possible but we also want to have as many
inputs as possible, sometimes inputs become available over time and we have to decide how long to wait.
Some tips for building models
● Build for data available at model call time/place
Figure out what data is actually available at the place in the system that is going to use our model
Figure out at what point in time the model is going to be called and what data we have at that point in time
Expect to have missing data at runtime and prepare for it
● Describe your inputs and outputs
When defining inputs have a description for each one that helps people figure out what it is
● Communicate early and optimize
Start a conversation with with people early in the process about what model needs to run so it can be planned for and be ready to
iterate on the inputs to adjust for the system limitations
Optimize for the way data is being processed batch or single piece at the time and make sure model is fast enough for the intended
use case
Some more tips for building models
● Build lookup tables for “static” data
Not everything has to be provided to the model, consider making lookup tables that contain some of the “static” slower changing
inputs. For example average user activity per day per state, or popularity per year, make, model, trim of a vehicle. Also include keys
for missing in the lookup tables.
Bundle lookup tables with the model and make an output of the training process. So each time a model is updated tables can be
updated as well.
● Retrain often and include previous model tracking
Automate model training process and retrain the model often if data is chaining.
Setup tracking of the model once it is in production and record inputs and outputs. Compare those values with what is expected and
include it in future model training.
● Be careful with string inputs data normalization might not happen on the source of data
and some edge case values might appear rarely enough to be missed. Define a set of
acceptable inputs or do normalization at runtime
How we host our
models
We support thousands of requests per second and
few tens of milliseconds response times in some
cases!
This is an example of how we could host a model
and does not have to represent a real vroom
approach.
● Iterate on model code with DS
owner many times
● Provide a template on how to
deliver and build a model
● Provide a stable, fast and
reliable platform that makes
calling models easy
● Provide an easy way to run many
version of the model and to add
new versions
● Be ready to evolve to satisfy
requirements and add new
features
Example of a hosted model
● Each time model is
trained a new mode data
version is created
● Model calling code is
shared for all model data
versions
● Service can load multiple
versions at the same time
● Model data version is
picked dynamically
● Alias can be defined and
pointed to specific model
data version
Simplified example system
How to define model code and model data
Model data
Produced by training, owned by DS, on remote storage
Model code
Owned by the ENG team
Located in the hosting
service
Consists of model artifact and lookup files
Artifact is packed up code (like .pkl file ) that has simple api to
call a predict function.
Lookup files should have values needed for input processing
and can be used by the model code and values from them are
passed in to model artifact
Should be updated often
New model data can be produced with each training cycle, API
and format should remain the same so it is compatible with
model code using it
Model code should not be
changed often (ever)
It is intended to do pre processing and post
processing of inputs/outputs using lookup
files
Model code should not contain
model logic
All the logic should be in model artifact
that is called by the model code.
How to integrate a model
● Write some code to invoke the trained model
Define generalized calling code that is parametrized by lookup files and knows how to pass in data and interpret the result. Make that
in to model code
● Add all dependencies
Identify all dependencies model needs to run, all inputs, all files and extract some inputs into lookup and make it in to model data
● Deliver model data to a remote location
Define a location where model data is delivered and the way data is structured
● iterate, iterate, iterate
Analyze performance and speed and iterate on improving it, convert from batch processing to single input processing, change data
types used or rework code blocks that are sub optimal
Promote model through environments and compared results with expected results
● Integrate with other service and start using it in production
What does a platform like this offer
● Easy and fast integration
Simple calling code logic to invoke the model written in python
● Easy and fast model updates
Model data is delivered to a remote location at will at any cadence.
Service automatically scans and detects new model data versions periodically at runtime.
● Easy and fast testing
Detected model data versions can be loaded(or unloaded) and hosted dynamically
Service hosts multiple model data versions of a model at the same time allowing for easy comparison, AB testing, and independent
updated of different use cases.
When calling the service a specific model data version that should be used can be defined.
If requested version is loaded response is sub second, if that version is discovered but not loaded then service downloads required
model data, loads it up and starts hosting it and then result is returned still within the same request with a delay. After first load
subsequent calls are fast.
What more does a platform like this offer
● Easy data manipulation
You can define calling code to transform your input before passing it to the model and also do the same thing with your outputs.
● Monitoring and constant uptime are built in
We monitor and report various metrics and have alerting for potential issues
● Designed for scaling and cost optimization
We can scale horizontally to cover huge loads and have automatic load based scaling and automatic unhealthy host replacements
with continuous uptime
We can host model with high resource requirements in an optimized way
We run efficiently with multiple model data versions sharing the same server but also multiple different models each having multiple
model data versions are sharing the same server.
This cuts down on cost dramatically and it might not seem as important with only one or two models in use, but when we get in to
tens of models each with many different model data versions that can become a huge cost if each model is running on a separate
server.
A peak under the
hood
● We use python based service for model hosting
to make DS integration of calling code easy
● We expose a set of REST API routes for each
supported model
● We run on aws ec2 instances directly and
support gpu and cpu based models
● We use s3 to store model data
● We run batch predictions and one by one real
time predictions and produce millions of
predictions per day
● Models are called by services that specialize in
orchestration, data gathering and caching
host/some-model/1/predict?modelDataVersion=last_one
Request:
[{
"vehicle": {
"year": 2019,
"make": "Lamborghini",
"model": "Huracan Spyder"
....
}
other inputs....
}]
Response:
{
"modelDataVersion": "2022-11-11",
"prediction": [
{
"reconditioningCost": "a lot of money :)"
}
]
}
*fictional api example
Let’s summarize
We also do a lot of other interesting stuff!
● Start building the pipeline early
in the proces
● Don’t build a model in a silo
● Ability to run many models in
parallel and many model
versions for each of them
● Automated support for updating
models
● Ability to run efficiently and
scale to match high loads
Vroom is hiring!
Reach out to our recruiters via LinkedIn to find out
more or send us your CV at vroomcareer@vast.com
Questions?
S3 structure example
Model loading example

Weitere ähnliche Inhalte

Ähnlich wie [DSC Europe 22] Engineers guide for shepherding models in to production - Marko Dimitrijevic

Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...
Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...
Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...Anant Corporation
 
SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!
SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!
SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!Richard Robinson
 
Scaling Ride-Hailing with Machine Learning on MLflow
Scaling Ride-Hailing with Machine Learning on MLflowScaling Ride-Hailing with Machine Learning on MLflow
Scaling Ride-Hailing with Machine Learning on MLflowDatabricks
 
Rsqrd AI: ML Tooling at an AI-first Startup
Rsqrd AI: ML Tooling at an AI-first StartupRsqrd AI: ML Tooling at an AI-first Startup
Rsqrd AI: ML Tooling at an AI-first StartupSanjana Chowdhury
 
Past Experiences and Future Challenges using Automatic Performance Modelling ...
Past Experiences and Future Challenges using Automatic Performance Modelling ...Past Experiences and Future Challenges using Automatic Performance Modelling ...
Past Experiences and Future Challenges using Automatic Performance Modelling ...Paul Brebner
 
Ml ops intro session
Ml ops   intro sessionMl ops   intro session
Ml ops intro sessionAvinash Patil
 
Cs 568 Spring 10 Lecture 5 Estimation
Cs 568 Spring 10  Lecture 5 EstimationCs 568 Spring 10  Lecture 5 Estimation
Cs 568 Spring 10 Lecture 5 EstimationLawrence Bernstein
 
Productionising Machine Learning Models
Productionising Machine Learning ModelsProductionising Machine Learning Models
Productionising Machine Learning ModelsTash Bickley
 
End to end MLworkflows
End to end MLworkflowsEnd to end MLworkflows
End to end MLworkflowsAdam Gibson
 
World Artificial Intelligence Conference Shanghai 2018
World Artificial Intelligence Conference Shanghai 2018World Artificial Intelligence Conference Shanghai 2018
World Artificial Intelligence Conference Shanghai 2018Adam Gibson
 
AI hype or reality
AI  hype or realityAI  hype or reality
AI hype or realityAwantik Das
 
Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...
Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...
Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...CodeScience
 
DevOps and Machine Learning (Geekwire Cloud Tech Summit)
DevOps and Machine Learning (Geekwire Cloud Tech Summit)DevOps and Machine Learning (Geekwire Cloud Tech Summit)
DevOps and Machine Learning (Geekwire Cloud Tech Summit)Jasjeet Thind
 
Managing the Complexities of Conversion to S1000D
Managing the Complexities of Conversion to S1000DManaging the Complexities of Conversion to S1000D
Managing the Complexities of Conversion to S1000Ddclsocialmedia
 
Machine Learning for Capacity Management
 Machine Learning for Capacity Management Machine Learning for Capacity Management
Machine Learning for Capacity ManagementEDB
 
StarWest 2019 - End to end testing: Stupid or Legit?
StarWest 2019 - End to end testing: Stupid or Legit?StarWest 2019 - End to end testing: Stupid or Legit?
StarWest 2019 - End to end testing: Stupid or Legit?mabl
 
Testing in the New World of Off-the-Shelf Software
Testing in the New World of Off-the-Shelf SoftwareTesting in the New World of Off-the-Shelf Software
Testing in the New World of Off-the-Shelf SoftwareJosiah Renaudin
 
Characteristics of modern data architecture that drive innovation
Characteristics of modern data architecture that drive innovationCharacteristics of modern data architecture that drive innovation
Characteristics of modern data architecture that drive innovationCloverDX
 
Multi Layer Federated Learning.pptx
Multi Layer Federated Learning.pptxMulti Layer Federated Learning.pptx
Multi Layer Federated Learning.pptxTimePass43152
 
Es_module2ppt.pptx
Es_module2ppt.pptxEs_module2ppt.pptx
Es_module2ppt.pptxRohanAM1
 

Ähnlich wie [DSC Europe 22] Engineers guide for shepherding models in to production - Marko Dimitrijevic (20)

Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...
Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...
Data Engineer's Lunch 89: Machine Learning Orchestration with AirflowMachine ...
 
SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!
SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!
SCRIMPS-STD: Test Automation Design Principles - and asking the right questions!
 
Scaling Ride-Hailing with Machine Learning on MLflow
Scaling Ride-Hailing with Machine Learning on MLflowScaling Ride-Hailing with Machine Learning on MLflow
Scaling Ride-Hailing with Machine Learning on MLflow
 
Rsqrd AI: ML Tooling at an AI-first Startup
Rsqrd AI: ML Tooling at an AI-first StartupRsqrd AI: ML Tooling at an AI-first Startup
Rsqrd AI: ML Tooling at an AI-first Startup
 
Past Experiences and Future Challenges using Automatic Performance Modelling ...
Past Experiences and Future Challenges using Automatic Performance Modelling ...Past Experiences and Future Challenges using Automatic Performance Modelling ...
Past Experiences and Future Challenges using Automatic Performance Modelling ...
 
Ml ops intro session
Ml ops   intro sessionMl ops   intro session
Ml ops intro session
 
Cs 568 Spring 10 Lecture 5 Estimation
Cs 568 Spring 10  Lecture 5 EstimationCs 568 Spring 10  Lecture 5 Estimation
Cs 568 Spring 10 Lecture 5 Estimation
 
Productionising Machine Learning Models
Productionising Machine Learning ModelsProductionising Machine Learning Models
Productionising Machine Learning Models
 
End to end MLworkflows
End to end MLworkflowsEnd to end MLworkflows
End to end MLworkflows
 
World Artificial Intelligence Conference Shanghai 2018
World Artificial Intelligence Conference Shanghai 2018World Artificial Intelligence Conference Shanghai 2018
World Artificial Intelligence Conference Shanghai 2018
 
AI hype or reality
AI  hype or realityAI  hype or reality
AI hype or reality
 
Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...
Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...
Technical Webinar: Patterns for Integrating Your Salesforce App with Off-Plat...
 
DevOps and Machine Learning (Geekwire Cloud Tech Summit)
DevOps and Machine Learning (Geekwire Cloud Tech Summit)DevOps and Machine Learning (Geekwire Cloud Tech Summit)
DevOps and Machine Learning (Geekwire Cloud Tech Summit)
 
Managing the Complexities of Conversion to S1000D
Managing the Complexities of Conversion to S1000DManaging the Complexities of Conversion to S1000D
Managing the Complexities of Conversion to S1000D
 
Machine Learning for Capacity Management
 Machine Learning for Capacity Management Machine Learning for Capacity Management
Machine Learning for Capacity Management
 
StarWest 2019 - End to end testing: Stupid or Legit?
StarWest 2019 - End to end testing: Stupid or Legit?StarWest 2019 - End to end testing: Stupid or Legit?
StarWest 2019 - End to end testing: Stupid or Legit?
 
Testing in the New World of Off-the-Shelf Software
Testing in the New World of Off-the-Shelf SoftwareTesting in the New World of Off-the-Shelf Software
Testing in the New World of Off-the-Shelf Software
 
Characteristics of modern data architecture that drive innovation
Characteristics of modern data architecture that drive innovationCharacteristics of modern data architecture that drive innovation
Characteristics of modern data architecture that drive innovation
 
Multi Layer Federated Learning.pptx
Multi Layer Federated Learning.pptxMulti Layer Federated Learning.pptx
Multi Layer Federated Learning.pptx
 
Es_module2ppt.pptx
Es_module2ppt.pptxEs_module2ppt.pptx
Es_module2ppt.pptx
 

Mehr von DataScienceConferenc1

[DSC Europe 23] Luciano Catani - AI in Diplomacy.PDF
[DSC Europe 23] Luciano Catani - AI in Diplomacy.PDF[DSC Europe 23] Luciano Catani - AI in Diplomacy.PDF
[DSC Europe 23] Luciano Catani - AI in Diplomacy.PDFDataScienceConferenc1
 
[DSC Europe 23] Rania Wazir - Mathematician jokes, cute cat photos, offensiv...
[DSC Europe 23] Rania Wazir -  Mathematician jokes, cute cat photos, offensiv...[DSC Europe 23] Rania Wazir -  Mathematician jokes, cute cat photos, offensiv...
[DSC Europe 23] Rania Wazir - Mathematician jokes, cute cat photos, offensiv...DataScienceConferenc1
 
[DSC Europe 23] Irena Cerovic - AI in International Development.pdf
[DSC Europe 23] Irena Cerovic - AI in International Development.pdf[DSC Europe 23] Irena Cerovic - AI in International Development.pdf
[DSC Europe 23] Irena Cerovic - AI in International Development.pdfDataScienceConferenc1
 
[DSC Europe 23] Ilija Duni - How Foursquare Builds Meaningful Bridges Between...
[DSC Europe 23] Ilija Duni - How Foursquare Builds Meaningful Bridges Between...[DSC Europe 23] Ilija Duni - How Foursquare Builds Meaningful Bridges Between...
[DSC Europe 23] Ilija Duni - How Foursquare Builds Meaningful Bridges Between...DataScienceConferenc1
 
[DSC Europe 23] Branka Panic - Peace in the age of artificial intelligence.pptx
[DSC Europe 23] Branka Panic - Peace in the age of artificial intelligence.pptx[DSC Europe 23] Branka Panic - Peace in the age of artificial intelligence.pptx
[DSC Europe 23] Branka Panic - Peace in the age of artificial intelligence.pptxDataScienceConferenc1
 
[DSC Europe 23][DigiHealth] Goran Dumic - Data-Driven Approach In Treatments
[DSC Europe 23][DigiHealth]  Goran Dumic -  Data-Driven Approach In Treatments[DSC Europe 23][DigiHealth]  Goran Dumic -  Data-Driven Approach In Treatments
[DSC Europe 23][DigiHealth] Goran Dumic - Data-Driven Approach In TreatmentsDataScienceConferenc1
 
[DSC Europe 23][DigiHealth] Milos Todorovic - Bridging the Gap-Innovating Ag...
[DSC Europe 23][DigiHealth]  Milos Todorovic - Bridging the Gap-Innovating Ag...[DSC Europe 23][DigiHealth]  Milos Todorovic - Bridging the Gap-Innovating Ag...
[DSC Europe 23][DigiHealth] Milos Todorovic - Bridging the Gap-Innovating Ag...DataScienceConferenc1
 
[DSC Europe 23][DigiHealth] Urosh VIlimanovich Clinical Data Management and C...
[DSC Europe 23][DigiHealth] Urosh VIlimanovich Clinical Data Management and C...[DSC Europe 23][DigiHealth] Urosh VIlimanovich Clinical Data Management and C...
[DSC Europe 23][DigiHealth] Urosh VIlimanovich Clinical Data Management and C...DataScienceConferenc1
 
[DSC Europe 23][DigiHealth] Vladimir Brusic - SMART HEALTH HOME: Technology,...
[DSC Europe 23][DigiHealth]  Vladimir Brusic - SMART HEALTH HOME: Technology,...[DSC Europe 23][DigiHealth]  Vladimir Brusic - SMART HEALTH HOME: Technology,...
[DSC Europe 23][DigiHealth] Vladimir Brusic - SMART HEALTH HOME: Technology,...DataScienceConferenc1
 
[DSC Europe 23][DigiHealth] Dimitar Penkov Grid Search Optimization of Novel...
[DSC Europe 23][DigiHealth]  Dimitar Penkov Grid Search Optimization of Novel...[DSC Europe 23][DigiHealth]  Dimitar Penkov Grid Search Optimization of Novel...
[DSC Europe 23][DigiHealth] Dimitar Penkov Grid Search Optimization of Novel...DataScienceConferenc1
 
[DSC Europe 23][DigiHealth] Tomislav Krizan - AIMED
[DSC Europe 23][DigiHealth] Tomislav Krizan - AIMED[DSC Europe 23][DigiHealth] Tomislav Krizan - AIMED
[DSC Europe 23][DigiHealth] Tomislav Krizan - AIMEDDataScienceConferenc1
 
[DSC Europe 23][DigiHealth] Katarina Vucicevic - Navigating theKinetics of Dr...
[DSC Europe 23][DigiHealth] Katarina Vucicevic - Navigating theKinetics of Dr...[DSC Europe 23][DigiHealth] Katarina Vucicevic - Navigating theKinetics of Dr...
[DSC Europe 23][DigiHealth] Katarina Vucicevic - Navigating theKinetics of Dr...DataScienceConferenc1
 
[DSC Europe 23][DigiHealth] Anja Baresic 0- Croatian digital Healthcare ecosy...
[DSC Europe 23][DigiHealth] Anja Baresic 0- Croatian digital Healthcare ecosy...[DSC Europe 23][DigiHealth] Anja Baresic 0- Croatian digital Healthcare ecosy...
[DSC Europe 23][DigiHealth] Anja Baresic 0- Croatian digital Healthcare ecosy...DataScienceConferenc1
 
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...
[DSC Europe 23][AI:CSI]  Dragan Pleskonjic - AI Impact on Cybersecurity and P...[DSC Europe 23][AI:CSI]  Dragan Pleskonjic - AI Impact on Cybersecurity and P...
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...DataScienceConferenc1
 
[DSC Europe 23][AI:CSI] Uros Arsenijevic Unlocking Cybersecurity with Seif
[DSC Europe 23][AI:CSI] Uros Arsenijevic Unlocking Cybersecurity with Seif[DSC Europe 23][AI:CSI] Uros Arsenijevic Unlocking Cybersecurity with Seif
[DSC Europe 23][AI:CSI] Uros Arsenijevic Unlocking Cybersecurity with SeifDataScienceConferenc1
 
[DSC Europe 23][AI:CSI] Goran Gvozden Improving Cybersecurity Posture with an...
[DSC Europe 23][AI:CSI] Goran Gvozden Improving Cybersecurity Posture with an...[DSC Europe 23][AI:CSI] Goran Gvozden Improving Cybersecurity Posture with an...
[DSC Europe 23][AI:CSI] Goran Gvozden Improving Cybersecurity Posture with an...DataScienceConferenc1
 
[DSC Europe 23][AI:CSI] Aleksa Stojanovic - Applying AI for Threat Detection ...
[DSC Europe 23][AI:CSI] Aleksa Stojanovic - Applying AI for Threat Detection ...[DSC Europe 23][AI:CSI] Aleksa Stojanovic - Applying AI for Threat Detection ...
[DSC Europe 23][AI:CSI] Aleksa Stojanovic - Applying AI for Threat Detection ...DataScienceConferenc1
 
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...DataScienceConferenc1
 
[DSC Europe 23][DigiHealth] Ligia Kornowska-How_may AI help you
[DSC Europe 23][DigiHealth] Ligia Kornowska-How_may AI help you[DSC Europe 23][DigiHealth] Ligia Kornowska-How_may AI help you
[DSC Europe 23][DigiHealth] Ligia Kornowska-How_may AI help youDataScienceConferenc1
 
[DSC Europe 23][DigiHealth] Ilya Zakharov - NETWORK NEUROSCIENCE WHERE THE BR...
[DSC Europe 23][DigiHealth] Ilya Zakharov - NETWORK NEUROSCIENCE WHERE THE BR...[DSC Europe 23][DigiHealth] Ilya Zakharov - NETWORK NEUROSCIENCE WHERE THE BR...
[DSC Europe 23][DigiHealth] Ilya Zakharov - NETWORK NEUROSCIENCE WHERE THE BR...DataScienceConferenc1
 

Mehr von DataScienceConferenc1 (20)

[DSC Europe 23] Luciano Catani - AI in Diplomacy.PDF
[DSC Europe 23] Luciano Catani - AI in Diplomacy.PDF[DSC Europe 23] Luciano Catani - AI in Diplomacy.PDF
[DSC Europe 23] Luciano Catani - AI in Diplomacy.PDF
 
[DSC Europe 23] Rania Wazir - Mathematician jokes, cute cat photos, offensiv...
[DSC Europe 23] Rania Wazir -  Mathematician jokes, cute cat photos, offensiv...[DSC Europe 23] Rania Wazir -  Mathematician jokes, cute cat photos, offensiv...
[DSC Europe 23] Rania Wazir - Mathematician jokes, cute cat photos, offensiv...
 
[DSC Europe 23] Irena Cerovic - AI in International Development.pdf
[DSC Europe 23] Irena Cerovic - AI in International Development.pdf[DSC Europe 23] Irena Cerovic - AI in International Development.pdf
[DSC Europe 23] Irena Cerovic - AI in International Development.pdf
 
[DSC Europe 23] Ilija Duni - How Foursquare Builds Meaningful Bridges Between...
[DSC Europe 23] Ilija Duni - How Foursquare Builds Meaningful Bridges Between...[DSC Europe 23] Ilija Duni - How Foursquare Builds Meaningful Bridges Between...
[DSC Europe 23] Ilija Duni - How Foursquare Builds Meaningful Bridges Between...
 
[DSC Europe 23] Branka Panic - Peace in the age of artificial intelligence.pptx
[DSC Europe 23] Branka Panic - Peace in the age of artificial intelligence.pptx[DSC Europe 23] Branka Panic - Peace in the age of artificial intelligence.pptx
[DSC Europe 23] Branka Panic - Peace in the age of artificial intelligence.pptx
 
[DSC Europe 23][DigiHealth] Goran Dumic - Data-Driven Approach In Treatments
[DSC Europe 23][DigiHealth]  Goran Dumic -  Data-Driven Approach In Treatments[DSC Europe 23][DigiHealth]  Goran Dumic -  Data-Driven Approach In Treatments
[DSC Europe 23][DigiHealth] Goran Dumic - Data-Driven Approach In Treatments
 
[DSC Europe 23][DigiHealth] Milos Todorovic - Bridging the Gap-Innovating Ag...
[DSC Europe 23][DigiHealth]  Milos Todorovic - Bridging the Gap-Innovating Ag...[DSC Europe 23][DigiHealth]  Milos Todorovic - Bridging the Gap-Innovating Ag...
[DSC Europe 23][DigiHealth] Milos Todorovic - Bridging the Gap-Innovating Ag...
 
[DSC Europe 23][DigiHealth] Urosh VIlimanovich Clinical Data Management and C...
[DSC Europe 23][DigiHealth] Urosh VIlimanovich Clinical Data Management and C...[DSC Europe 23][DigiHealth] Urosh VIlimanovich Clinical Data Management and C...
[DSC Europe 23][DigiHealth] Urosh VIlimanovich Clinical Data Management and C...
 
[DSC Europe 23][DigiHealth] Vladimir Brusic - SMART HEALTH HOME: Technology,...
[DSC Europe 23][DigiHealth]  Vladimir Brusic - SMART HEALTH HOME: Technology,...[DSC Europe 23][DigiHealth]  Vladimir Brusic - SMART HEALTH HOME: Technology,...
[DSC Europe 23][DigiHealth] Vladimir Brusic - SMART HEALTH HOME: Technology,...
 
[DSC Europe 23][DigiHealth] Dimitar Penkov Grid Search Optimization of Novel...
[DSC Europe 23][DigiHealth]  Dimitar Penkov Grid Search Optimization of Novel...[DSC Europe 23][DigiHealth]  Dimitar Penkov Grid Search Optimization of Novel...
[DSC Europe 23][DigiHealth] Dimitar Penkov Grid Search Optimization of Novel...
 
[DSC Europe 23][DigiHealth] Tomislav Krizan - AIMED
[DSC Europe 23][DigiHealth] Tomislav Krizan - AIMED[DSC Europe 23][DigiHealth] Tomislav Krizan - AIMED
[DSC Europe 23][DigiHealth] Tomislav Krizan - AIMED
 
[DSC Europe 23][DigiHealth] Katarina Vucicevic - Navigating theKinetics of Dr...
[DSC Europe 23][DigiHealth] Katarina Vucicevic - Navigating theKinetics of Dr...[DSC Europe 23][DigiHealth] Katarina Vucicevic - Navigating theKinetics of Dr...
[DSC Europe 23][DigiHealth] Katarina Vucicevic - Navigating theKinetics of Dr...
 
[DSC Europe 23][DigiHealth] Anja Baresic 0- Croatian digital Healthcare ecosy...
[DSC Europe 23][DigiHealth] Anja Baresic 0- Croatian digital Healthcare ecosy...[DSC Europe 23][DigiHealth] Anja Baresic 0- Croatian digital Healthcare ecosy...
[DSC Europe 23][DigiHealth] Anja Baresic 0- Croatian digital Healthcare ecosy...
 
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...
[DSC Europe 23][AI:CSI]  Dragan Pleskonjic - AI Impact on Cybersecurity and P...[DSC Europe 23][AI:CSI]  Dragan Pleskonjic - AI Impact on Cybersecurity and P...
[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...
 
[DSC Europe 23][AI:CSI] Uros Arsenijevic Unlocking Cybersecurity with Seif
[DSC Europe 23][AI:CSI] Uros Arsenijevic Unlocking Cybersecurity with Seif[DSC Europe 23][AI:CSI] Uros Arsenijevic Unlocking Cybersecurity with Seif
[DSC Europe 23][AI:CSI] Uros Arsenijevic Unlocking Cybersecurity with Seif
 
[DSC Europe 23][AI:CSI] Goran Gvozden Improving Cybersecurity Posture with an...
[DSC Europe 23][AI:CSI] Goran Gvozden Improving Cybersecurity Posture with an...[DSC Europe 23][AI:CSI] Goran Gvozden Improving Cybersecurity Posture with an...
[DSC Europe 23][AI:CSI] Goran Gvozden Improving Cybersecurity Posture with an...
 
[DSC Europe 23][AI:CSI] Aleksa Stojanovic - Applying AI for Threat Detection ...
[DSC Europe 23][AI:CSI] Aleksa Stojanovic - Applying AI for Threat Detection ...[DSC Europe 23][AI:CSI] Aleksa Stojanovic - Applying AI for Threat Detection ...
[DSC Europe 23][AI:CSI] Aleksa Stojanovic - Applying AI for Threat Detection ...
 
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...
[DSC Europe 23][DigiHealth] Muthu Ramachandran AI and Blockchain Framework fo...
 
[DSC Europe 23][DigiHealth] Ligia Kornowska-How_may AI help you
[DSC Europe 23][DigiHealth] Ligia Kornowska-How_may AI help you[DSC Europe 23][DigiHealth] Ligia Kornowska-How_may AI help you
[DSC Europe 23][DigiHealth] Ligia Kornowska-How_may AI help you
 
[DSC Europe 23][DigiHealth] Ilya Zakharov - NETWORK NEUROSCIENCE WHERE THE BR...
[DSC Europe 23][DigiHealth] Ilya Zakharov - NETWORK NEUROSCIENCE WHERE THE BR...[DSC Europe 23][DigiHealth] Ilya Zakharov - NETWORK NEUROSCIENCE WHERE THE BR...
[DSC Europe 23][DigiHealth] Ilya Zakharov - NETWORK NEUROSCIENCE WHERE THE BR...
 

Kürzlich hochgeladen

FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024
FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024
FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024Susanna-Assunta Sansone
 
What To Do For World Nature Conservation Day by Slidesgo.pptx
What To Do For World Nature Conservation Day by Slidesgo.pptxWhat To Do For World Nature Conservation Day by Slidesgo.pptx
What To Do For World Nature Conservation Day by Slidesgo.pptxSimranPal17
 
Cyber awareness ppt on the recorded data
Cyber awareness ppt on the recorded dataCyber awareness ppt on the recorded data
Cyber awareness ppt on the recorded dataTecnoIncentive
 
Decoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis ProjectDecoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis ProjectBoston Institute of Analytics
 
SMOTE and K-Fold Cross Validation-Presentation.pptx
SMOTE and K-Fold Cross Validation-Presentation.pptxSMOTE and K-Fold Cross Validation-Presentation.pptx
SMOTE and K-Fold Cross Validation-Presentation.pptxHaritikaChhatwal1
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksdeepakthakur548787
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaManalVerma4
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Boston Institute of Analytics
 
Student Profile Sample report on improving academic performance by uniting gr...
Student Profile Sample report on improving academic performance by uniting gr...Student Profile Sample report on improving academic performance by uniting gr...
Student Profile Sample report on improving academic performance by uniting gr...Seán Kennedy
 
Rithik Kumar Singh codealpha pythohn.pdf
Rithik Kumar Singh codealpha pythohn.pdfRithik Kumar Singh codealpha pythohn.pdf
Rithik Kumar Singh codealpha pythohn.pdfrahulyadav957181
 
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelDecoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelBoston Institute of Analytics
 
Real-Time AI Streaming - AI Max Princeton
Real-Time AI  Streaming - AI Max PrincetonReal-Time AI  Streaming - AI Max Princeton
Real-Time AI Streaming - AI Max PrincetonTimothy Spann
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen
 
Learn How Data Science Changes Our World
Learn How Data Science Changes Our WorldLearn How Data Science Changes Our World
Learn How Data Science Changes Our WorldEduminds Learning
 
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Thomas Poetter
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Boston Institute of Analytics
 
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Boston Institute of Analytics
 
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdfWorld Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdfsimulationsindia
 

Kürzlich hochgeladen (20)

FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024
FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024
FAIR, FAIRsharing, FAIR Cookbook and ELIXIR - Sansone SA - Boston 2024
 
What To Do For World Nature Conservation Day by Slidesgo.pptx
What To Do For World Nature Conservation Day by Slidesgo.pptxWhat To Do For World Nature Conservation Day by Slidesgo.pptx
What To Do For World Nature Conservation Day by Slidesgo.pptx
 
Cyber awareness ppt on the recorded data
Cyber awareness ppt on the recorded dataCyber awareness ppt on the recorded data
Cyber awareness ppt on the recorded data
 
Decoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis ProjectDecoding Patterns: Customer Churn Prediction Data Analysis Project
Decoding Patterns: Customer Churn Prediction Data Analysis Project
 
SMOTE and K-Fold Cross Validation-Presentation.pptx
SMOTE and K-Fold Cross Validation-Presentation.pptxSMOTE and K-Fold Cross Validation-Presentation.pptx
SMOTE and K-Fold Cross Validation-Presentation.pptx
 
Digital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing worksDigital Marketing Plan, how digital marketing works
Digital Marketing Plan, how digital marketing works
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in India
 
Data Analysis Project: Stroke Prediction
Data Analysis Project: Stroke PredictionData Analysis Project: Stroke Prediction
Data Analysis Project: Stroke Prediction
 
Insurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis ProjectInsurance Churn Prediction Data Analysis Project
Insurance Churn Prediction Data Analysis Project
 
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...
 
Student Profile Sample report on improving academic performance by uniting gr...
Student Profile Sample report on improving academic performance by uniting gr...Student Profile Sample report on improving academic performance by uniting gr...
Student Profile Sample report on improving academic performance by uniting gr...
 
Rithik Kumar Singh codealpha pythohn.pdf
Rithik Kumar Singh codealpha pythohn.pdfRithik Kumar Singh codealpha pythohn.pdf
Rithik Kumar Singh codealpha pythohn.pdf
 
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis modelDecoding Movie Sentiments: Analyzing Reviews with Data Analysis model
Decoding Movie Sentiments: Analyzing Reviews with Data Analysis model
 
Real-Time AI Streaming - AI Max Princeton
Real-Time AI  Streaming - AI Max PrincetonReal-Time AI  Streaming - AI Max Princeton
Real-Time AI Streaming - AI Max Princeton
 
Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)Data Factory in Microsoft Fabric (MsBIP #82)
Data Factory in Microsoft Fabric (MsBIP #82)
 
Learn How Data Science Changes Our World
Learn How Data Science Changes Our WorldLearn How Data Science Changes Our World
Learn How Data Science Changes Our World
 
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
Minimizing AI Hallucinations/Confabulations and the Path towards AGI with Exa...
 
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...
 
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
Data Analysis Project : Targeting the Right Customers, Presentation on Bank M...
 
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdfWorld Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
World Economic Forum Metaverse Ecosystem By Utpal Chakraborty.pdf
 

[DSC Europe 22] Engineers guide for shepherding models in to production - Marko Dimitrijevic

  • 1. Engineers guide for shepherding models in to production Marko Dimitrijević Staff software engineer at vroom marko.dimitrijevic@vast.com m_a_r_e.91@hotmail.com
  • 2. What is an example of a data science model? ● Takes in data ● Learns from data ● Gets updated ● Produces results ● Provides value
  • 3. What does it mean to run a model in production environment And what does it mean to run a model in real time ● Returns a result every time (or a meaningful error) ● Inputs and outputs are well defined and explained ● Performant and reasonably optimized for the task ● Results can be depended on and are verified to be correct ● System is able to handle errors and edge cases
  • 4. Two main topics we will cover Building model pipeline ● Defining inputs and outputs ● Defining a call place ● Collecting inputs and delivering outputs ● Handling edge cases Hosting the model ● Defining resource needs ● Optimizing model code ● Integrating with platform code ● Testing and iterating
  • 5. What does the engineering team do We also rename stuff to follow standards! ● We take care of monitoring and scaling to match the load ● We take care of the model hosting and plumbing for providing inputs and outputs ● We also advise on what is doable and not and help find the optimal solution ● We help rework models to run faster, better or to follow good practices ● We know to ask the right question and prevent future problems
  • 6. How model pipeline is built Assuming research part is done and algorithm is picked ● Produce the model ● Identify where we can get the data for training ● Find a way to get all the inputs the model needs ● Find the right location to integrate the model and call it ● Deliver model outputs to the right place
  • 7. When model trains on processed data Not a good way to do it!
  • 8. Common issues in building model pipelines Most common issues are data related ● Model is using data upstream of the place from where it will be called. This can lead to inputs that do not take raw data but some processed/aggregated version of it that is not available at model call place. ● Data gets renamed many times on its way from source to the model training Sometimes people call same things different names and vice versa, this can lead to confusion and wrong inputs being used. If i want a location is that location of the customer or vehicle location or address of reconditioning center handling the vehicle? ● Incomplete data is filtered out in training but can it be in real world It is easy to ignore incomplete data when there is a lot of leftover data to work with, but in the real world pieces of information are often missing and sometimes a prediction is better than no prediction. Coverage can be very low for models that are strict about their inputs.
  • 9. Common issues in building model pipelines Potential timing issues ● If model is called in real time and user is waiting for a response it should be <1s Sometimes models are not built to operate fast on a single input but are optimized for batch processing. Also we often need to spend time querying different services to prepare model inputs and that time adds up to the total response time to the user. ● Once model is done and needs to be integrated the process takes a long time Inserting a model in a production flow can take time and if we need to build a pipeline that gets multiple different and “exotic” inputs that can include many teams and be time consuming ● It is hard to identify the right moment in time to trigger the model We often want to predict something as early in the process as possible but we also want to have as many inputs as possible, sometimes inputs become available over time and we have to decide how long to wait.
  • 10. Some tips for building models ● Build for data available at model call time/place Figure out what data is actually available at the place in the system that is going to use our model Figure out at what point in time the model is going to be called and what data we have at that point in time Expect to have missing data at runtime and prepare for it ● Describe your inputs and outputs When defining inputs have a description for each one that helps people figure out what it is ● Communicate early and optimize Start a conversation with with people early in the process about what model needs to run so it can be planned for and be ready to iterate on the inputs to adjust for the system limitations Optimize for the way data is being processed batch or single piece at the time and make sure model is fast enough for the intended use case
  • 11. Some more tips for building models ● Build lookup tables for “static” data Not everything has to be provided to the model, consider making lookup tables that contain some of the “static” slower changing inputs. For example average user activity per day per state, or popularity per year, make, model, trim of a vehicle. Also include keys for missing in the lookup tables. Bundle lookup tables with the model and make an output of the training process. So each time a model is updated tables can be updated as well. ● Retrain often and include previous model tracking Automate model training process and retrain the model often if data is chaining. Setup tracking of the model once it is in production and record inputs and outputs. Compare those values with what is expected and include it in future model training. ● Be careful with string inputs data normalization might not happen on the source of data and some edge case values might appear rarely enough to be missed. Define a set of acceptable inputs or do normalization at runtime
  • 12. How we host our models We support thousands of requests per second and few tens of milliseconds response times in some cases! This is an example of how we could host a model and does not have to represent a real vroom approach. ● Iterate on model code with DS owner many times ● Provide a template on how to deliver and build a model ● Provide a stable, fast and reliable platform that makes calling models easy ● Provide an easy way to run many version of the model and to add new versions ● Be ready to evolve to satisfy requirements and add new features
  • 13. Example of a hosted model ● Each time model is trained a new mode data version is created ● Model calling code is shared for all model data versions ● Service can load multiple versions at the same time ● Model data version is picked dynamically ● Alias can be defined and pointed to specific model data version Simplified example system
  • 14. How to define model code and model data Model data Produced by training, owned by DS, on remote storage Model code Owned by the ENG team Located in the hosting service Consists of model artifact and lookup files Artifact is packed up code (like .pkl file ) that has simple api to call a predict function. Lookup files should have values needed for input processing and can be used by the model code and values from them are passed in to model artifact Should be updated often New model data can be produced with each training cycle, API and format should remain the same so it is compatible with model code using it Model code should not be changed often (ever) It is intended to do pre processing and post processing of inputs/outputs using lookup files Model code should not contain model logic All the logic should be in model artifact that is called by the model code.
  • 15. How to integrate a model ● Write some code to invoke the trained model Define generalized calling code that is parametrized by lookup files and knows how to pass in data and interpret the result. Make that in to model code ● Add all dependencies Identify all dependencies model needs to run, all inputs, all files and extract some inputs into lookup and make it in to model data ● Deliver model data to a remote location Define a location where model data is delivered and the way data is structured ● iterate, iterate, iterate Analyze performance and speed and iterate on improving it, convert from batch processing to single input processing, change data types used or rework code blocks that are sub optimal Promote model through environments and compared results with expected results ● Integrate with other service and start using it in production
  • 16. What does a platform like this offer ● Easy and fast integration Simple calling code logic to invoke the model written in python ● Easy and fast model updates Model data is delivered to a remote location at will at any cadence. Service automatically scans and detects new model data versions periodically at runtime. ● Easy and fast testing Detected model data versions can be loaded(or unloaded) and hosted dynamically Service hosts multiple model data versions of a model at the same time allowing for easy comparison, AB testing, and independent updated of different use cases. When calling the service a specific model data version that should be used can be defined. If requested version is loaded response is sub second, if that version is discovered but not loaded then service downloads required model data, loads it up and starts hosting it and then result is returned still within the same request with a delay. After first load subsequent calls are fast.
  • 17. What more does a platform like this offer ● Easy data manipulation You can define calling code to transform your input before passing it to the model and also do the same thing with your outputs. ● Monitoring and constant uptime are built in We monitor and report various metrics and have alerting for potential issues ● Designed for scaling and cost optimization We can scale horizontally to cover huge loads and have automatic load based scaling and automatic unhealthy host replacements with continuous uptime We can host model with high resource requirements in an optimized way We run efficiently with multiple model data versions sharing the same server but also multiple different models each having multiple model data versions are sharing the same server. This cuts down on cost dramatically and it might not seem as important with only one or two models in use, but when we get in to tens of models each with many different model data versions that can become a huge cost if each model is running on a separate server.
  • 18. A peak under the hood ● We use python based service for model hosting to make DS integration of calling code easy ● We expose a set of REST API routes for each supported model ● We run on aws ec2 instances directly and support gpu and cpu based models ● We use s3 to store model data ● We run batch predictions and one by one real time predictions and produce millions of predictions per day ● Models are called by services that specialize in orchestration, data gathering and caching host/some-model/1/predict?modelDataVersion=last_one Request: [{ "vehicle": { "year": 2019, "make": "Lamborghini", "model": "Huracan Spyder" .... } other inputs.... }] Response: { "modelDataVersion": "2022-11-11", "prediction": [ { "reconditioningCost": "a lot of money :)" } ] } *fictional api example
  • 19. Let’s summarize We also do a lot of other interesting stuff! ● Start building the pipeline early in the proces ● Don’t build a model in a silo ● Ability to run many models in parallel and many model versions for each of them ● Automated support for updating models ● Ability to run efficiently and scale to match high loads
  • 20. Vroom is hiring! Reach out to our recruiters via LinkedIn to find out more or send us your CV at vroomcareer@vast.com