SlideShare ist ein Scribd-Unternehmen logo
1 von 35
Downloaden Sie, um offline zu lesen
Accelerating Model Development by
Reducing Operational Barriers
Patrick Hayes, Cofounder & CTO, SigOpt
Talk ID: S9556
Accelerate and amplify the
impact of modelers everywhere
3
Hardware Environment
SigOpt automates experimentation and optimization
Transformation
Labeling
Pre-Processing
Pipeline Dev.
Feature Eng.
Feature Stores
Data
Preparation
Experimentation, Training, Evaluation
Notebook & Model Framework
Experimentation & Model Optimization
On-Premise Hybrid Multi-Cloud
Insights, Tracking,
Collaboration
Model Search,
Hyperparameter Tuning
Resource Scheduler,
Management
Validation
Serving
Deploying
Monitoring
Managing
Inference
Online Testing
Model
Deployment
Hyperparameter Optimization
Model Tuning
Grid Search
Random Search Bayesian Optimization
Training & Tuning
Evolutionary Algorithms
Deep Learning Architecture Search
Hyperparameter Search
How it works: Seamlessly tune any model
ML, DL or
Simulation Model
Model Evaluation or
Backtest
Testing
Data
Training
Data
Never
accesses your
data or models
How it Works: Seamless implementation for any stack
Install SigOpt1
Create experiment2
Parameterize model3
Run optimization loop4
Analyze experiments5
How it Works: Seamless implementation for any stack
Install SigOpt1
Create experiment2
Parameterize model3
Run optimization loop4
Analyze experiments5
https://bit.ly/sigopt-notebook
How it Works: Seamless implementation for any stack
Install SigOpt1
Create experiment2
Parameterize model3
Run optimization loop4
Analyze experiments5
https://bit.ly/sigopt-notebook
How it Works: Seamless implementation for any stack
Install SigOpt1
Create experiment2
Parameterize model3
Run optimization loop4
Analyze experiments5
https://bit.ly/sigopt-notebook
How it Works: Seamless implementation for any stack
Install SigOpt1
Create experiment2
Parameterize model3
Run optimization loop4
Analyze experiments5
https://bit.ly/sigopt-notebook
How it Works: Seamless implementation for any stack
Install SigOpt1
Create experiment2
Parameterize model3
Run optimization loop4
Analyze experiments5
13
Benefits: Better, Cheaper, Faster Model Development
90% Cost Savings
Maximize utilization of compute
https://aws.amazon.com/blogs/machine-learning/fast-
cnn-tuning-with-aws-gpu-instances-and-sigopt/
10x Faster Time to Tune
Less expert time per model
https://devblogs.nvidia.com/sigopt-deep-learning-hyp
erparameter-optimization/
Better Performance
No free lunch, but optimize any model
https://arxiv.org/pdf/1603.09441.pdf
14
Overview of Features Behind SigOpt
Enterprise
Platform
Optimization
Engine
Experiment
Insights
Reproducibility
Intuitive web dashboards
Cross-team permissions
and collaboration
Advanced experiment
visualizations
Organizational
experiment analysis
Parameter importance
analysis
Multimetric optimization
Continuous, categorical,
or integer parameters
Constraints and failure
regions
Up to 10k observations,
100 parameters
Multitask optimization
and high parallelism
Conditional
parameters
Infrastructure agnostic
REST API
Model agnostic
Black-box interface
Doesn’t touch data
Libraries for Python,
Java, R, and MATLAB
Key:
Only HPO solution
with this capability
Applied deep learning introduces
unique challenges
Failed observations
Constraints
Uncertainty
Competing objectives
Lengthy training cycles
Cluster orchestration
sigopt.com/blog
How do you more efficiently tune models
that take days (or weeks) to train?
18
AlexNex to AlphaGo Zero: 300,000x Increase in Compute
2012 20192013 2014 2015 2016 2017 2018
.00001
10,000
1
Petaflop/s-Day(Training)
Year
• AlexNet
• Dropout
• Visualizing and Understanding Conv Nets
• DQN
• GoogleNet
• DeepSpeech2
• ResNets
• Xception
• Neural Architecture Search
• Neural Machine Translation
• AlphaZero
• AlphaGo Zero
• TI7 Dota 1v1
VGG
• Seq2Seq
19
Speech Recognition Deep Reinforcement Learning Computer Vision
Training Resnet-50 on ImageNet takes 10 hours
Tuning 12 parameters requires at least 120 distinct models
That equals 1,200 hours, or 50 days, of training time
Running optimization tasks in parallel is critical to tuning
expensive deep learning models
Multiple Users
Concurrent Optimization Experiments
Concurrent Model Configuration Evaluations
Multiple GPUs per Model
22
Complexity of Deep Learning DevOps
Training One Model, No Optimization
Basic Case Advanced Case
23
Cluster Orchestration
1 Spin up and share training clusters Schedule optimization experiments2
Integrate with the optimization API3 Monitor experiment and infrastructure4
Problems:
Infrastructure, scheduling,
dependencies, code, monitoring
Solution:
SigOpt Orchestrate is a CLI for
managing training infrastructure and
running optimization experiments
How it Works
Seamless Integration into Your Model Code
Easily Define Optimization Experiments
Easily Kick Off Optimization Experiment Jobs
Check the Status of Active and Completed Experiments
View Experiment Logs Across Multiple Workers
Track Metadata and Monitor Your Results
Automated Cluster Management
Training Resnet-50 on ImageNet takes 10 hours
Tuning 12 parameters requires at least 120 distinct models
That equals 1,200 hours, or 50 days, of training time
While training on 20 machines, wall-clock time is 50 days 2.5 days
Failed Observations
Constraints
Uncertainty
Competing Objectives
Lengthy Training Cycles
Cluster Orchestration
sigopt.com/blog
Thank you!
patrick@sigopt.com for additional questions.
Try SigOpt Orchestrate: https://sigopt.com/orchestrate
Free access for Academics & Nonprofits: https://sigopt.com/edu
Solution-oriented program for the Enterprise: https://sigopt.com/pricing
Leading applied optimization research: https://sigopt.com/research
… and we're hiring! https://sigopt.com/careers

Weitere ähnliche Inhalte

Ähnlich wie SigOpt at GTC - Reducing operational barriers to optimization

Productionizing Machine Learning with a Microservices Architecture
Productionizing Machine Learning with a Microservices ArchitectureProductionizing Machine Learning with a Microservices Architecture
Productionizing Machine Learning with a Microservices ArchitectureDatabricks
 
Advanced MLflow: Multi-Step Workflows, Hyperparameter Tuning and Integrating ...
Advanced MLflow: Multi-Step Workflows, Hyperparameter Tuning and Integrating ...Advanced MLflow: Multi-Step Workflows, Hyperparameter Tuning and Integrating ...
Advanced MLflow: Multi-Step Workflows, Hyperparameter Tuning and Integrating ...Databricks
 
Danny Bickson - Python based predictive analytics with GraphLab Create
Danny Bickson - Python based predictive analytics with GraphLab Create Danny Bickson - Python based predictive analytics with GraphLab Create
Danny Bickson - Python based predictive analytics with GraphLab Create PyData
 
GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...
GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...
GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...James Anderson
 
Jonathon Wright - Intelligent Performance Cognitive Learning (AIOps)
Jonathon Wright - Intelligent Performance Cognitive Learning (AIOps)Jonathon Wright - Intelligent Performance Cognitive Learning (AIOps)
Jonathon Wright - Intelligent Performance Cognitive Learning (AIOps)Neotys_Partner
 
Trenowanie i wdrażanie modeli uczenia maszynowego z wykorzystaniem Google Clo...
Trenowanie i wdrażanie modeli uczenia maszynowego z wykorzystaniem Google Clo...Trenowanie i wdrażanie modeli uczenia maszynowego z wykorzystaniem Google Clo...
Trenowanie i wdrażanie modeli uczenia maszynowego z wykorzystaniem Google Clo...Sotrender
 
Peek into Neo4j Product Strategy and Roadmap
Peek into Neo4j Product Strategy and RoadmapPeek into Neo4j Product Strategy and Roadmap
Peek into Neo4j Product Strategy and RoadmapNeo4j
 
Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...
Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...
Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...MLconf
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareJustin Basilico
 
World Artificial Intelligence Conference Shanghai 2018
World Artificial Intelligence Conference Shanghai 2018World Artificial Intelligence Conference Shanghai 2018
World Artificial Intelligence Conference Shanghai 2018Adam Gibson
 
Machine Learning Infrastructure
Machine Learning InfrastructureMachine Learning Infrastructure
Machine Learning InfrastructureSigOpt
 
Machine Learning and AI
Machine Learning and AIMachine Learning and AI
Machine Learning and AIJames Serra
 
DevOps and Machine Learning (Geekwire Cloud Tech Summit)
DevOps and Machine Learning (Geekwire Cloud Tech Summit)DevOps and Machine Learning (Geekwire Cloud Tech Summit)
DevOps and Machine Learning (Geekwire Cloud Tech Summit)Jasjeet Thind
 
SigOpt for Machine Learning and AI
SigOpt for Machine Learning and AISigOpt for Machine Learning and AI
SigOpt for Machine Learning and AISigOpt
 
Microsoft DevOps for AI with GoDataDriven
Microsoft DevOps for AI with GoDataDrivenMicrosoft DevOps for AI with GoDataDriven
Microsoft DevOps for AI with GoDataDrivenGoDataDriven
 
Serverless machine learning architectures at Helixa
Serverless machine learning architectures at HelixaServerless machine learning architectures at Helixa
Serverless machine learning architectures at HelixaData Science Milan
 
B4UConference_machine learning_deeplearning
B4UConference_machine learning_deeplearningB4UConference_machine learning_deeplearning
B4UConference_machine learning_deeplearningHoa Le
 
Tuning 2.0: Advanced Optimization Techniques Webinar
Tuning 2.0: Advanced Optimization Techniques WebinarTuning 2.0: Advanced Optimization Techniques Webinar
Tuning 2.0: Advanced Optimization Techniques WebinarSigOpt
 
201908 Overview of Automated ML
201908 Overview of Automated ML201908 Overview of Automated ML
201908 Overview of Automated MLMark Tabladillo
 

Ähnlich wie SigOpt at GTC - Reducing operational barriers to optimization (20)

Productionizing Machine Learning with a Microservices Architecture
Productionizing Machine Learning with a Microservices ArchitectureProductionizing Machine Learning with a Microservices Architecture
Productionizing Machine Learning with a Microservices Architecture
 
Advanced MLflow: Multi-Step Workflows, Hyperparameter Tuning and Integrating ...
Advanced MLflow: Multi-Step Workflows, Hyperparameter Tuning and Integrating ...Advanced MLflow: Multi-Step Workflows, Hyperparameter Tuning and Integrating ...
Advanced MLflow: Multi-Step Workflows, Hyperparameter Tuning and Integrating ...
 
Danny Bickson - Python based predictive analytics with GraphLab Create
Danny Bickson - Python based predictive analytics with GraphLab Create Danny Bickson - Python based predictive analytics with GraphLab Create
Danny Bickson - Python based predictive analytics with GraphLab Create
 
GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...
GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...
GDG Cloud Southlake #16: Priyanka Vergadia: Scalable Data Analytics in Google...
 
Jonathon Wright - Intelligent Performance Cognitive Learning (AIOps)
Jonathon Wright - Intelligent Performance Cognitive Learning (AIOps)Jonathon Wright - Intelligent Performance Cognitive Learning (AIOps)
Jonathon Wright - Intelligent Performance Cognitive Learning (AIOps)
 
Trenowanie i wdrażanie modeli uczenia maszynowego z wykorzystaniem Google Clo...
Trenowanie i wdrażanie modeli uczenia maszynowego z wykorzystaniem Google Clo...Trenowanie i wdrażanie modeli uczenia maszynowego z wykorzystaniem Google Clo...
Trenowanie i wdrażanie modeli uczenia maszynowego z wykorzystaniem Google Clo...
 
Peek into Neo4j Product Strategy and Roadmap
Peek into Neo4j Product Strategy and RoadmapPeek into Neo4j Product Strategy and Roadmap
Peek into Neo4j Product Strategy and Roadmap
 
Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...
Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...
Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning Software
 
World Artificial Intelligence Conference Shanghai 2018
World Artificial Intelligence Conference Shanghai 2018World Artificial Intelligence Conference Shanghai 2018
World Artificial Intelligence Conference Shanghai 2018
 
Machine Learning Infrastructure
Machine Learning InfrastructureMachine Learning Infrastructure
Machine Learning Infrastructure
 
Machine Learning and AI
Machine Learning and AIMachine Learning and AI
Machine Learning and AI
 
DevOps and Machine Learning (Geekwire Cloud Tech Summit)
DevOps and Machine Learning (Geekwire Cloud Tech Summit)DevOps and Machine Learning (Geekwire Cloud Tech Summit)
DevOps and Machine Learning (Geekwire Cloud Tech Summit)
 
SigOpt for Machine Learning and AI
SigOpt for Machine Learning and AISigOpt for Machine Learning and AI
SigOpt for Machine Learning and AI
 
Microsoft DevOps for AI with GoDataDriven
Microsoft DevOps for AI with GoDataDrivenMicrosoft DevOps for AI with GoDataDriven
Microsoft DevOps for AI with GoDataDriven
 
Serverless machine learning architectures at Helixa
Serverless machine learning architectures at HelixaServerless machine learning architectures at Helixa
Serverless machine learning architectures at Helixa
 
B4UConference_machine learning_deeplearning
B4UConference_machine learning_deeplearningB4UConference_machine learning_deeplearning
B4UConference_machine learning_deeplearning
 
Tuning 2.0: Advanced Optimization Techniques Webinar
Tuning 2.0: Advanced Optimization Techniques WebinarTuning 2.0: Advanced Optimization Techniques Webinar
Tuning 2.0: Advanced Optimization Techniques Webinar
 
MLOps in action
MLOps in actionMLOps in action
MLOps in action
 
201908 Overview of Automated ML
201908 Overview of Automated ML201908 Overview of Automated ML
201908 Overview of Automated ML
 

Mehr von SigOpt

Optimizing BERT and Natural Language Models with SigOpt Experiment Management
Optimizing BERT and Natural Language Models with SigOpt Experiment ManagementOptimizing BERT and Natural Language Models with SigOpt Experiment Management
Optimizing BERT and Natural Language Models with SigOpt Experiment ManagementSigOpt
 
Experiment Management for the Enterprise
Experiment Management for the EnterpriseExperiment Management for the Enterprise
Experiment Management for the EnterpriseSigOpt
 
Efficient NLP by Distilling BERT and Multimetric Optimization
Efficient NLP by Distilling BERT and Multimetric OptimizationEfficient NLP by Distilling BERT and Multimetric Optimization
Efficient NLP by Distilling BERT and Multimetric OptimizationSigOpt
 
Detecting COVID-19 Cases with Deep Learning
Detecting COVID-19 Cases with Deep LearningDetecting COVID-19 Cases with Deep Learning
Detecting COVID-19 Cases with Deep LearningSigOpt
 
Metric Management: a SigOpt Applied Use Case
Metric Management: a SigOpt Applied Use CaseMetric Management: a SigOpt Applied Use Case
Metric Management: a SigOpt Applied Use CaseSigOpt
 
Tuning Data Augmentation to Boost Model Performance
Tuning Data Augmentation to Boost Model PerformanceTuning Data Augmentation to Boost Model Performance
Tuning Data Augmentation to Boost Model PerformanceSigOpt
 
Advanced Optimization for the Enterprise Webinar
Advanced Optimization for the Enterprise WebinarAdvanced Optimization for the Enterprise Webinar
Advanced Optimization for the Enterprise WebinarSigOpt
 
Modeling at Scale: SigOpt at TWIMLcon 2019
Modeling at Scale: SigOpt at TWIMLcon 2019Modeling at Scale: SigOpt at TWIMLcon 2019
Modeling at Scale: SigOpt at TWIMLcon 2019SigOpt
 
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...SigOpt
 
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...SigOpt
 
SigOpt at GTC - Tuning the Untunable
SigOpt at GTC - Tuning the UntunableSigOpt at GTC - Tuning the Untunable
SigOpt at GTC - Tuning the UntunableSigOpt
 
Lessons for an enterprise approach to modeling at scale
Lessons for an enterprise approach to modeling at scaleLessons for an enterprise approach to modeling at scale
Lessons for an enterprise approach to modeling at scaleSigOpt
 
SigOpt at MLconf - Reducing Operational Barriers to Model Training
SigOpt at MLconf - Reducing Operational Barriers to Model TrainingSigOpt at MLconf - Reducing Operational Barriers to Model Training
SigOpt at MLconf - Reducing Operational Barriers to Model TrainingSigOpt
 
Machine Learning Infrastructure
Machine Learning InfrastructureMachine Learning Infrastructure
Machine Learning InfrastructureSigOpt
 
Tips and techniques for hyperparameter optimization
Tips and techniques for hyperparameter optimizationTips and techniques for hyperparameter optimization
Tips and techniques for hyperparameter optimizationSigOpt
 
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...SigOpt
 
Using Optimal Learning to Tune Deep Learning Pipelines
Using Optimal Learning to Tune Deep Learning PipelinesUsing Optimal Learning to Tune Deep Learning Pipelines
Using Optimal Learning to Tune Deep Learning PipelinesSigOpt
 
Common Problems in Hyperparameter Optimization
Common Problems in Hyperparameter OptimizationCommon Problems in Hyperparameter Optimization
Common Problems in Hyperparameter OptimizationSigOpt
 
SigOpt for Hedge Funds
SigOpt for Hedge FundsSigOpt for Hedge Funds
SigOpt for Hedge FundsSigOpt
 

Mehr von SigOpt (19)

Optimizing BERT and Natural Language Models with SigOpt Experiment Management
Optimizing BERT and Natural Language Models with SigOpt Experiment ManagementOptimizing BERT and Natural Language Models with SigOpt Experiment Management
Optimizing BERT and Natural Language Models with SigOpt Experiment Management
 
Experiment Management for the Enterprise
Experiment Management for the EnterpriseExperiment Management for the Enterprise
Experiment Management for the Enterprise
 
Efficient NLP by Distilling BERT and Multimetric Optimization
Efficient NLP by Distilling BERT and Multimetric OptimizationEfficient NLP by Distilling BERT and Multimetric Optimization
Efficient NLP by Distilling BERT and Multimetric Optimization
 
Detecting COVID-19 Cases with Deep Learning
Detecting COVID-19 Cases with Deep LearningDetecting COVID-19 Cases with Deep Learning
Detecting COVID-19 Cases with Deep Learning
 
Metric Management: a SigOpt Applied Use Case
Metric Management: a SigOpt Applied Use CaseMetric Management: a SigOpt Applied Use Case
Metric Management: a SigOpt Applied Use Case
 
Tuning Data Augmentation to Boost Model Performance
Tuning Data Augmentation to Boost Model PerformanceTuning Data Augmentation to Boost Model Performance
Tuning Data Augmentation to Boost Model Performance
 
Advanced Optimization for the Enterprise Webinar
Advanced Optimization for the Enterprise WebinarAdvanced Optimization for the Enterprise Webinar
Advanced Optimization for the Enterprise Webinar
 
Modeling at Scale: SigOpt at TWIMLcon 2019
Modeling at Scale: SigOpt at TWIMLcon 2019Modeling at Scale: SigOpt at TWIMLcon 2019
Modeling at Scale: SigOpt at TWIMLcon 2019
 
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...
 
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...
 
SigOpt at GTC - Tuning the Untunable
SigOpt at GTC - Tuning the UntunableSigOpt at GTC - Tuning the Untunable
SigOpt at GTC - Tuning the Untunable
 
Lessons for an enterprise approach to modeling at scale
Lessons for an enterprise approach to modeling at scaleLessons for an enterprise approach to modeling at scale
Lessons for an enterprise approach to modeling at scale
 
SigOpt at MLconf - Reducing Operational Barriers to Model Training
SigOpt at MLconf - Reducing Operational Barriers to Model TrainingSigOpt at MLconf - Reducing Operational Barriers to Model Training
SigOpt at MLconf - Reducing Operational Barriers to Model Training
 
Machine Learning Infrastructure
Machine Learning InfrastructureMachine Learning Infrastructure
Machine Learning Infrastructure
 
Tips and techniques for hyperparameter optimization
Tips and techniques for hyperparameter optimizationTips and techniques for hyperparameter optimization
Tips and techniques for hyperparameter optimization
 
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...
 
Using Optimal Learning to Tune Deep Learning Pipelines
Using Optimal Learning to Tune Deep Learning PipelinesUsing Optimal Learning to Tune Deep Learning Pipelines
Using Optimal Learning to Tune Deep Learning Pipelines
 
Common Problems in Hyperparameter Optimization
Common Problems in Hyperparameter OptimizationCommon Problems in Hyperparameter Optimization
Common Problems in Hyperparameter Optimization
 
SigOpt for Hedge Funds
SigOpt for Hedge FundsSigOpt for Hedge Funds
SigOpt for Hedge Funds
 

Kürzlich hochgeladen

AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.Sharon Liu
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 

Kürzlich hochgeladen (20)

AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 

SigOpt at GTC - Reducing operational barriers to optimization

  • 1. Accelerating Model Development by Reducing Operational Barriers Patrick Hayes, Cofounder & CTO, SigOpt Talk ID: S9556
  • 2. Accelerate and amplify the impact of modelers everywhere
  • 3. 3
  • 4. Hardware Environment SigOpt automates experimentation and optimization Transformation Labeling Pre-Processing Pipeline Dev. Feature Eng. Feature Stores Data Preparation Experimentation, Training, Evaluation Notebook & Model Framework Experimentation & Model Optimization On-Premise Hybrid Multi-Cloud Insights, Tracking, Collaboration Model Search, Hyperparameter Tuning Resource Scheduler, Management Validation Serving Deploying Monitoring Managing Inference Online Testing Model Deployment
  • 5. Hyperparameter Optimization Model Tuning Grid Search Random Search Bayesian Optimization Training & Tuning Evolutionary Algorithms Deep Learning Architecture Search Hyperparameter Search
  • 6. How it works: Seamlessly tune any model ML, DL or Simulation Model Model Evaluation or Backtest Testing Data Training Data Never accesses your data or models
  • 7. How it Works: Seamless implementation for any stack Install SigOpt1 Create experiment2 Parameterize model3 Run optimization loop4 Analyze experiments5
  • 8. How it Works: Seamless implementation for any stack Install SigOpt1 Create experiment2 Parameterize model3 Run optimization loop4 Analyze experiments5 https://bit.ly/sigopt-notebook
  • 9. How it Works: Seamless implementation for any stack Install SigOpt1 Create experiment2 Parameterize model3 Run optimization loop4 Analyze experiments5 https://bit.ly/sigopt-notebook
  • 10. How it Works: Seamless implementation for any stack Install SigOpt1 Create experiment2 Parameterize model3 Run optimization loop4 Analyze experiments5 https://bit.ly/sigopt-notebook
  • 11. How it Works: Seamless implementation for any stack Install SigOpt1 Create experiment2 Parameterize model3 Run optimization loop4 Analyze experiments5 https://bit.ly/sigopt-notebook
  • 12. How it Works: Seamless implementation for any stack Install SigOpt1 Create experiment2 Parameterize model3 Run optimization loop4 Analyze experiments5
  • 13. 13 Benefits: Better, Cheaper, Faster Model Development 90% Cost Savings Maximize utilization of compute https://aws.amazon.com/blogs/machine-learning/fast- cnn-tuning-with-aws-gpu-instances-and-sigopt/ 10x Faster Time to Tune Less expert time per model https://devblogs.nvidia.com/sigopt-deep-learning-hyp erparameter-optimization/ Better Performance No free lunch, but optimize any model https://arxiv.org/pdf/1603.09441.pdf
  • 14. 14 Overview of Features Behind SigOpt Enterprise Platform Optimization Engine Experiment Insights Reproducibility Intuitive web dashboards Cross-team permissions and collaboration Advanced experiment visualizations Organizational experiment analysis Parameter importance analysis Multimetric optimization Continuous, categorical, or integer parameters Constraints and failure regions Up to 10k observations, 100 parameters Multitask optimization and high parallelism Conditional parameters Infrastructure agnostic REST API Model agnostic Black-box interface Doesn’t touch data Libraries for Python, Java, R, and MATLAB Key: Only HPO solution with this capability
  • 15. Applied deep learning introduces unique challenges
  • 16. Failed observations Constraints Uncertainty Competing objectives Lengthy training cycles Cluster orchestration sigopt.com/blog
  • 17. How do you more efficiently tune models that take days (or weeks) to train?
  • 18. 18 AlexNex to AlphaGo Zero: 300,000x Increase in Compute 2012 20192013 2014 2015 2016 2017 2018 .00001 10,000 1 Petaflop/s-Day(Training) Year • AlexNet • Dropout • Visualizing and Understanding Conv Nets • DQN • GoogleNet • DeepSpeech2 • ResNets • Xception • Neural Architecture Search • Neural Machine Translation • AlphaZero • AlphaGo Zero • TI7 Dota 1v1 VGG • Seq2Seq
  • 19. 19 Speech Recognition Deep Reinforcement Learning Computer Vision
  • 20. Training Resnet-50 on ImageNet takes 10 hours Tuning 12 parameters requires at least 120 distinct models That equals 1,200 hours, or 50 days, of training time
  • 21. Running optimization tasks in parallel is critical to tuning expensive deep learning models
  • 22. Multiple Users Concurrent Optimization Experiments Concurrent Model Configuration Evaluations Multiple GPUs per Model 22 Complexity of Deep Learning DevOps Training One Model, No Optimization Basic Case Advanced Case
  • 23. 23 Cluster Orchestration 1 Spin up and share training clusters Schedule optimization experiments2 Integrate with the optimization API3 Monitor experiment and infrastructure4
  • 24. Problems: Infrastructure, scheduling, dependencies, code, monitoring Solution: SigOpt Orchestrate is a CLI for managing training infrastructure and running optimization experiments
  • 26. Seamless Integration into Your Model Code
  • 28. Easily Kick Off Optimization Experiment Jobs
  • 29. Check the Status of Active and Completed Experiments
  • 30. View Experiment Logs Across Multiple Workers
  • 31. Track Metadata and Monitor Your Results
  • 33. Training Resnet-50 on ImageNet takes 10 hours Tuning 12 parameters requires at least 120 distinct models That equals 1,200 hours, or 50 days, of training time While training on 20 machines, wall-clock time is 50 days 2.5 days
  • 34. Failed Observations Constraints Uncertainty Competing Objectives Lengthy Training Cycles Cluster Orchestration sigopt.com/blog
  • 35. Thank you! patrick@sigopt.com for additional questions. Try SigOpt Orchestrate: https://sigopt.com/orchestrate Free access for Academics & Nonprofits: https://sigopt.com/edu Solution-oriented program for the Enterprise: https://sigopt.com/pricing Leading applied optimization research: https://sigopt.com/research … and we're hiring! https://sigopt.com/careers