SlideShare ist ein Scribd-Unternehmen logo
1 von 38
Downloaden Sie, um offline zu lesen
SigOpt. Confidential.
Talk #1
Intuition behind Bayesian optimization
with and without multiple metrics
SigOpt Talk Series
Tuning for Systematic Trading
Tobias Andreasen — Machine Learning Engineer
Tuesday, March 24, 2020
SigOpt. Confidential.
Abstract
SigOpt provides an extensive set of advanced features,
which help you, the expert, save time while
increasing model performance via experimentation.
Today, we will start out this talking series by giving an
overview of general black box optimization, efficient
bayesian optimization and end up extending this to
multiple competing metrics.
SigOpt. Confidential.
Motivation
1. Overview of SigOpt
2. How to solve a black box optimization problem
3. Why you should optimize using
multiple competing metrics
SigOpt. Confidential.
Overview of SigOpt1
SigOpt. Confidential.
Accelerate and amplify the
impact of modelers everywhere
SigOpt. Confidential.
Experiment Insights Optimization Engine
Track, analyze and reproduce any
model to improve the productivity of
your modeling
Enterprise Platform
Automate hyperparameter tuning to
maximize the performance and impact
of your models
Standardize experimentation across
any combination of library,
infrastructure, model or task
On-Premise Hybrid/Multi
Solution: Experiment, optimize and analyze at scale
6
SigOpt. Confidential.
SigOpt Features
Enterprise
Platform
Optimization
Engine
Experiment
Insights
Reproducibility
Intuitive web dashboards
Cross-team permissions
and collaboration
Advanced experiment
visualizations
Usage insights
Parameter importance
analysis
Multimetric optimization
Continuous, categorical, or
integer parameters
Constraints and
failure regions
Up to 10k observations,
100 parameters
Multitask optimization and
high parallelism
Conditional
parameters
Infrastructure agnostic
REST API
Parallel Resource Scheduler
Black-Box Interface
Tunes without
accessing any data
Libraries for Python,
Java, R, and MATLAB
SigOpt. Confidential.
How to solve a
black box optimization problem2
SigOpt. Confidential.
Why black box optimization?
SigOpt was designed to empower you, the practitioner, to re-define most machine
learning problems as black box optimization problems with the benefit of:
• Amplified performance — incremental gains in accuracy or other success metrics
• Productivity gains — a consistent platform across tasks that facilitates sharing
• Accelerated modeling — early elimination of non-scalable tasks
• Compute efficiency — continuous, full utilization of infrastructure
SigOpt uses an ensemble of Bayesian and Global Optimization methods to solve
these black box optimization problems.
9
Black Box Optimization
SigOpt. Confidential.
Your firewall
Training
Data
AI, ML, DL,
Simulation Model
Model Evaluation
or Backtest
Testing
Data
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
Black Box Optimization
SigOpt. Confidential.
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Black Box Optimization
Better
Results
SigOpt. Confidential.
Hyperparameter Optimization
Model Tuning
Grid Search
Random Search Bayesian Optimization
Training & Tuning
Evolutionary Algorithms
Deep Learning Architecture Search
Hyperparameter Search
SigOpt. Confidential.
Pro Con
Manual Search Leverages expertise Not scalable, inconsistent
Grid Search Simple to implement Not scalable, often infeasible
Random Search Scalable Inefficient
Evolutionary Algorithms Effective at architecture search Very resource intensive
Bayesian Optimization Efficient, effective Can be tough to parallelize
SigOpt. Confidential.
Pro Con
Manual Search Leverages expertise Not scalable, inconsistent
Grid Search Simple to implement Not scalable, often infeasible
Random Search Scalable Inefficient
Evolutionary Algorithms Effective at architecture search Very resource intensive
Bayesian Optimization Efficient, effective Can be tough to parallelize
SigOpt. Confidential.
Pro Con
Manual Search Leverages expertise Not scalable, inconsistent
Grid Search Simple to implement Not scalable, often infeasible
Random Search Scalable Inefficient
Evolutionary Algorithms Effective at architecture search Very resource intensive
Bayesian Optimization Efficient, effective Can be tough to parallelize
SigOpt. Confidential.
Pro Con
Manual Search Leverages expertise Not scalable, inconsistent
Grid Search Simple to implement Not scalable, often infeasible
Random Search Scalable Inefficient
Evolutionary Algorithms Effective at architecture search Very resource intensive
Bayesian Optimization Efficient, effective Can be tough to parallelize
SigOpt. Confidential.
Pro Con
Manual Search Leverages expertise Not scalable, inconsistent
Grid Search Simple to implement Not scalable, often infeasible
Random Search Scalable Inefficient
Evolutionary Algorithms Effective at architecture search Very resource intensive
Bayesian Optimization Efficient, effective Can be tough to parallelize
SigOpt. Confidential.
A graphical depiction of the iterative process
18
Build a statistical model
Sequential Model Based Optimization (SMBO)
SigOpt. Confidential.
A graphical depiction of the iterative process
19
Build a statistical model
Choose the next point
to maximize the acquisition function
Sequential Model Based Optimization (SMBO)
SigOpt. Confidential.
A graphical depiction of the iterative process
20
Build a statistical model Build a statistical model
Choose the next point
to maximize the acquisition function
Sequential Model Based Optimization (SMBO)
SigOpt. Confidential.
A graphical depiction of the iterative process
21
Build a statistical model Build a statistical model
Choose the next point
to maximize the acquisition function
Sequential Model Based Optimization (SMBO)
Choose the next point
to maximize the acquisition function
SigOpt. Confidential.
Gaussian processes: a powerful tool for modeling in spatial statistics
A standard tool for building statistical models is the Gaussian process
[Fasshauer et al, 2015, Fraizer, 2018].
• Assume that function values are jointly normally distributed.
• Apply prior beliefs about mean behavior and covariance between observations.
• Posterior beliefs about unobserved locations can be computed rather easily.
Different prior assumptions produce different statistical models:
22
Complexity #1: Optimizing the Model in SMBO
SigOpt. Confidential.
Acquisition function: given a model, how should we choose the next point?
An acquisition function is a strategy for defining the utility of a future sample, given the current samples, while
balancing exploration and exploitation [Shahriari et al, 2016].
Different acquisition functions choose different points (EI, PI, KG, etc.).
23
Complexity #2: Optimizing the SMBO Process
Exploration:
Learning about the whole function f
Exploitation:
Further resolving regions where good f values
have already been observed
SigOpt. Confidential.
RESTAPI
The SigOpt API handles the complexity
Better
Results
SigOpt Blog Posts: Intuition Behind Bayesian Optimization
Some Relevant Blog Posts
● Intuition Behind Covariance Kernels
● Approximation of Data
● Likelihood for Gaussian Processes
● Profile Likelihood vs. Kriging Variance
● Intuition behind Gaussian Processes
● Dealing with Troublesome Metrics
Find more blog posts visit:
https://sigopt.com/blog/
SigOpt. Confidential.
Why you should optimize using
multiple competing metrics
3
SigOpt. Confidential.
Why optimize against multiple competing metrics?
SigOpt allows the user to specify multiple competing metrics for either optimization or tracking to better
align success of the experimentation with business value:
• Multiple metrics — The option to defining multiple metrics, which can yield new and interesting results
• Insights, metric storage — Insights through tracking of optimized and unoptimized metrics
• Thresholds — The ability to define thresholds for success to better guide the optimizer
This process gives models that deliver more reliable business outcomes by helping optimally make the
tradeoffs inherent in the modeling process and real world applications.
27
Optimizing Multiple Competing Metrics
SigOpt. Confidential.
Your firewall
Training
Data
AI, ML, DL,
Simulation Models
Model Evaluation
or Backtest
Testing
Data
New Configurations
Objective
Metric
EXPERIMENT INSIGHTS
Track, organize, analyze and reproduce
any model
ENTERPRISE PLATFORM
Built to fit any stack and scale with your
needs
OPTIMIZATION ENGINE
Explore and exploit with a variety of
techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
Optimizing Multiple Competing Metrics
Better
Results
SigOpt. Confidential.
EXPERIMENT INSIGHTS
Track, organize, analyze and reproduce
any model
ENTERPRISE PLATFORM
Built to fit any stack and scale with your
needs
OPTIMIZATION ENGINE
Explore and exploit with a variety of
techniques
RESTAPI
Optimizing Multiple Competing Metrics
Multiple Optimized
and Unoptimized
Metrics
Better
Results
SigOpt. Confidential.
Looking for the right Balance
Balancing competing metrics to find the Pareto frontier
Most problems of practical relevance involve 2 or more competing metrics.
• Neural networks — Balancing accuracy and inference time
• Materials design — Balancing performance and maintenance cost
• Algo trading — Balancing Sharpe Ratio and book size
In a situation with Competing Metrics, the set of all efficient points (the Pareto frontier) is the solution.
30
Pareto
Frontier
Feasible
Region
SigOpt. Confidential.
Balancing competing metrics to find the Pareto frontier
As shown before, the goal in multi objective or multi criteria optimization the goal is to find the optimal set
of solution across a set of function [Knowles, 2006].
• This is formulated as finding the maximum of the set functions f1
to fn
over the same domain x
• No single point exist as the solution, but we are actively trying to maximize the size of the efficient
frontier, which represent the set of solutions
• The solution is found through scalarization methods such as convex combination and
epsilon-constraint
31
Finding the Best Tradeoffs
SigOpt. Confidential.
Optimizing Multiple Competing Metrics: First Approach
Intuition: Convex Combination Scalarization
Idea: If we can convert the multimetric problem into a scalar problem, we can solve this problem using
Bayesian optimization.
One possible scalarization is through a convex combination of the objectives.
32
SigOpt. Confidential.
Balancing competing metrics to find the Pareto frontier with threshold
As shown before, the goal in multi objective or multi criteria optimization the goal is to find the optimal set
of solution across a set of function [Knowles, 2006].
• This is formulated as finding the maximum of the set functions f1
to fn
over the same domain x
• No single point exist as the solution, but we are actively trying to maximize the size of the efficient
frontier, which represent the set of solutions
• The solution is found through constrained scalarization methods such as convex combination and
epsilon-constraint
• Allow users to change constraints as the search progresses [Letham et al, 2019]
33
Optimizing Multiple Competing Metrics: Deeper Approach
SigOpt. Confidential.
Rephrase the Entire Problem
Constrained Scalarization
1. Model all metrics independently.
• Requires no prior beliefs of how metrics interact.
• Missing data removed on a per metric basis if unrecorded.
2. Expose the efficient frontier through constrained scalar optimization.
• Enforce user constraints when given.
• Iterate through sub constraints to better resolve efficient frontier, if desired.
• Consider different regions of the frontier when parallelism is possible.
3. Allow users to change constraints as the search progresses.
• Allow the problems/goals to evolve as the user’s understanding changes.
Constraints give customers more control over the circumstances and more ability to understand our actions.
34
Variation on
Expected
Improvement
[Letham et al, 2019]
SigOpt. Confidential.
Intuition: Scalarization and Epsilon Constraints
35
Use Epsilon Constraints to Focus Search
SigOpt. Confidential.
Intuition: Constrained Scalarization and Epsilon Constraints
36
Add Thresholds to Limit Frontier
Multimetric Use Case 1
● Category: Time Series
● Task: Sequence Classification
● Model: CNN
● Data: Diatom Images
● Analysis: Accuracy-Time Tradeoff
● Result: Similar accuracy, 33% the inference time
Multimetric Use Case 2
● Category: NLP
● Task: Sentiment Analysis
● Model: CNN
● Data: Rotten Tomatoes Movie Reviews
● Analysis: Accuracy-Time Tradeoff
● Result: ~2% in accuracy versus 50% of training time
Learn more
https://devblogs.nvidia.com/sigopt-deep-learning-
hyperparameter-optimization/
Use Case: Balancing Speed & Accuracy in Deep Learning
SigOpt. Confidential.
Tobias Andreasen | tobias@sigopt.com
For more information visit: https://sigopt.com/research/
Questions?

Weitere ähnliche Inhalte

Was ist angesagt?

Common Problems in Hyperparameter Optimization
Common Problems in Hyperparameter OptimizationCommon Problems in Hyperparameter Optimization
Common Problems in Hyperparameter OptimizationSigOpt
 
Machine Learning Fundamentals
Machine Learning FundamentalsMachine Learning Fundamentals
Machine Learning FundamentalsSigOpt
 
MLOps Using MLflow
MLOps Using MLflowMLOps Using MLflow
MLOps Using MLflowDatabricks
 
Pentaho World 2017: Automated Machine Learning (AutoML) and Pentaho (Thursday...
Pentaho World 2017: Automated Machine Learning (AutoML) and Pentaho (Thursday...Pentaho World 2017: Automated Machine Learning (AutoML) and Pentaho (Thursday...
Pentaho World 2017: Automated Machine Learning (AutoML) and Pentaho (Thursday...Caio Moreno
 
PyData London 2018 talk on feature selection
PyData London 2018 talk on feature selectionPyData London 2018 talk on feature selection
PyData London 2018 talk on feature selectionThomas Huijskens
 
Model-Based Optimization for Effective and Reliable Decision-Making
Model-Based Optimization for Effective and Reliable Decision-MakingModel-Based Optimization for Effective and Reliable Decision-Making
Model-Based Optimization for Effective and Reliable Decision-MakingBob Fourer
 
Agile Machine Learning for Real-time Recommender Systems
Agile Machine Learning for Real-time Recommender SystemsAgile Machine Learning for Real-time Recommender Systems
Agile Machine Learning for Real-time Recommender SystemsJohann Schleier-Smith
 
Shortening the time from analysis to deployment with ml as-a-service — Luiz A...
Shortening the time from analysis to deployment with ml as-a-service — Luiz A...Shortening the time from analysis to deployment with ml as-a-service — Luiz A...
Shortening the time from analysis to deployment with ml as-a-service — Luiz A...PAPIs.io
 
“Houston, we have a model...” Introduction to MLOps
“Houston, we have a model...” Introduction to MLOps“Houston, we have a model...” Introduction to MLOps
“Houston, we have a model...” Introduction to MLOpsRui Quintino
 
Nesma autumn conference 2015 - Is FPA a valuable addition to predictable agil...
Nesma autumn conference 2015 - Is FPA a valuable addition to predictable agil...Nesma autumn conference 2015 - Is FPA a valuable addition to predictable agil...
Nesma autumn conference 2015 - Is FPA a valuable addition to predictable agil...Nesma
 
Recent advances on large scheduling problems in CP Optimizer
Recent advances on large scheduling problems in CP OptimizerRecent advances on large scheduling problems in CP Optimizer
Recent advances on large scheduling problems in CP OptimizerPhilippe Laborie
 
Model-Based Optimization / CP 2018
Model-Based Optimization / CP 2018Model-Based Optimization / CP 2018
Model-Based Optimization / CP 2018Bob Fourer
 

Was ist angesagt? (13)

Common Problems in Hyperparameter Optimization
Common Problems in Hyperparameter OptimizationCommon Problems in Hyperparameter Optimization
Common Problems in Hyperparameter Optimization
 
Machine Learning Fundamentals
Machine Learning FundamentalsMachine Learning Fundamentals
Machine Learning Fundamentals
 
MLOps Using MLflow
MLOps Using MLflowMLOps Using MLflow
MLOps Using MLflow
 
Pydata presentation
Pydata presentationPydata presentation
Pydata presentation
 
Pentaho World 2017: Automated Machine Learning (AutoML) and Pentaho (Thursday...
Pentaho World 2017: Automated Machine Learning (AutoML) and Pentaho (Thursday...Pentaho World 2017: Automated Machine Learning (AutoML) and Pentaho (Thursday...
Pentaho World 2017: Automated Machine Learning (AutoML) and Pentaho (Thursday...
 
PyData London 2018 talk on feature selection
PyData London 2018 talk on feature selectionPyData London 2018 talk on feature selection
PyData London 2018 talk on feature selection
 
Model-Based Optimization for Effective and Reliable Decision-Making
Model-Based Optimization for Effective and Reliable Decision-MakingModel-Based Optimization for Effective and Reliable Decision-Making
Model-Based Optimization for Effective and Reliable Decision-Making
 
Agile Machine Learning for Real-time Recommender Systems
Agile Machine Learning for Real-time Recommender SystemsAgile Machine Learning for Real-time Recommender Systems
Agile Machine Learning for Real-time Recommender Systems
 
Shortening the time from analysis to deployment with ml as-a-service — Luiz A...
Shortening the time from analysis to deployment with ml as-a-service — Luiz A...Shortening the time from analysis to deployment with ml as-a-service — Luiz A...
Shortening the time from analysis to deployment with ml as-a-service — Luiz A...
 
“Houston, we have a model...” Introduction to MLOps
“Houston, we have a model...” Introduction to MLOps“Houston, we have a model...” Introduction to MLOps
“Houston, we have a model...” Introduction to MLOps
 
Nesma autumn conference 2015 - Is FPA a valuable addition to predictable agil...
Nesma autumn conference 2015 - Is FPA a valuable addition to predictable agil...Nesma autumn conference 2015 - Is FPA a valuable addition to predictable agil...
Nesma autumn conference 2015 - Is FPA a valuable addition to predictable agil...
 
Recent advances on large scheduling problems in CP Optimizer
Recent advances on large scheduling problems in CP OptimizerRecent advances on large scheduling problems in CP Optimizer
Recent advances on large scheduling problems in CP Optimizer
 
Model-Based Optimization / CP 2018
Model-Based Optimization / CP 2018Model-Based Optimization / CP 2018
Model-Based Optimization / CP 2018
 

Ähnlich wie Tuning for Systematic Trading: Talk 1

Tuning for Systematic Trading: Talk 2: Deep Learning
Tuning for Systematic Trading: Talk 2: Deep LearningTuning for Systematic Trading: Talk 2: Deep Learning
Tuning for Systematic Trading: Talk 2: Deep LearningSigOpt
 
Modeling at Scale: SigOpt at TWIMLcon 2019
Modeling at Scale: SigOpt at TWIMLcon 2019Modeling at Scale: SigOpt at TWIMLcon 2019
Modeling at Scale: SigOpt at TWIMLcon 2019SigOpt
 
Tuning 2.0: Advanced Optimization Techniques Webinar
Tuning 2.0: Advanced Optimization Techniques WebinarTuning 2.0: Advanced Optimization Techniques Webinar
Tuning 2.0: Advanced Optimization Techniques WebinarSigOpt
 
Metric Management: a SigOpt Applied Use Case
Metric Management: a SigOpt Applied Use CaseMetric Management: a SigOpt Applied Use Case
Metric Management: a SigOpt Applied Use CaseSigOpt
 
SigOpt for Machine Learning and AI
SigOpt for Machine Learning and AISigOpt for Machine Learning and AI
SigOpt for Machine Learning and AISigOpt
 
Using Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning ModelsUsing Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning ModelsScott Clark
 
Using Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning ModelsUsing Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning ModelsSigOpt
 
DutchMLSchool. ML for Energy Trading and Automotive Sector
DutchMLSchool. ML for Energy Trading and Automotive SectorDutchMLSchool. ML for Energy Trading and Automotive Sector
DutchMLSchool. ML for Energy Trading and Automotive SectorBigML, Inc
 
Scaling & Transforming Stitch Fix's Visibility into What Folks will love
Scaling & Transforming Stitch Fix's Visibility into What Folks will loveScaling & Transforming Stitch Fix's Visibility into What Folks will love
Scaling & Transforming Stitch Fix's Visibility into What Folks will loveJune Andrews
 
SigOpt at GTC - Tuning the Untunable
SigOpt at GTC - Tuning the UntunableSigOpt at GTC - Tuning the Untunable
SigOpt at GTC - Tuning the UntunableSigOpt
 
Experiment Management for the Enterprise
Experiment Management for the EnterpriseExperiment Management for the Enterprise
Experiment Management for the EnterpriseSigOpt
 
Machine Learning/ Data Science: Boosting Predictive Analytics Model Performance
Machine Learning/ Data Science: Boosting Predictive Analytics Model PerformanceMachine Learning/ Data Science: Boosting Predictive Analytics Model Performance
Machine Learning/ Data Science: Boosting Predictive Analytics Model PerformanceT. Scott Clendaniel
 
MLConf 2016 SigOpt Talk by Scott Clark
MLConf 2016 SigOpt Talk by Scott ClarkMLConf 2016 SigOpt Talk by Scott Clark
MLConf 2016 SigOpt Talk by Scott ClarkSigOpt
 
Scott Clark, Co-Founder and CEO, SigOpt at MLconf SF 2016
Scott Clark, Co-Founder and CEO, SigOpt at MLconf SF 2016Scott Clark, Co-Founder and CEO, SigOpt at MLconf SF 2016
Scott Clark, Co-Founder and CEO, SigOpt at MLconf SF 2016MLconf
 
Failure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature DeliveryFailure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature DeliveryOptimizely
 
How to Lower the Cost of Deploying Analytics: An Introduction to the Portable...
How to Lower the Cost of Deploying Analytics: An Introduction to the Portable...How to Lower the Cost of Deploying Analytics: An Introduction to the Portable...
How to Lower the Cost of Deploying Analytics: An Introduction to the Portable...Robert Grossman
 
From notebook to production with Amazon Sagemaker
From notebook to production with Amazon SagemakerFrom notebook to production with Amazon Sagemaker
From notebook to production with Amazon SagemakerAmazon Web Services
 
Visualizing Model Selection with Scikit-Yellowbrick: An Introduction to Devel...
Visualizing Model Selection with Scikit-Yellowbrick: An Introduction to Devel...Visualizing Model Selection with Scikit-Yellowbrick: An Introduction to Devel...
Visualizing Model Selection with Scikit-Yellowbrick: An Introduction to Devel...Benjamin Bengfort
 

Ähnlich wie Tuning for Systematic Trading: Talk 1 (20)

Tuning for Systematic Trading: Talk 2: Deep Learning
Tuning for Systematic Trading: Talk 2: Deep LearningTuning for Systematic Trading: Talk 2: Deep Learning
Tuning for Systematic Trading: Talk 2: Deep Learning
 
Modeling at Scale: SigOpt at TWIMLcon 2019
Modeling at Scale: SigOpt at TWIMLcon 2019Modeling at Scale: SigOpt at TWIMLcon 2019
Modeling at Scale: SigOpt at TWIMLcon 2019
 
Tuning 2.0: Advanced Optimization Techniques Webinar
Tuning 2.0: Advanced Optimization Techniques WebinarTuning 2.0: Advanced Optimization Techniques Webinar
Tuning 2.0: Advanced Optimization Techniques Webinar
 
Metric Management: a SigOpt Applied Use Case
Metric Management: a SigOpt Applied Use CaseMetric Management: a SigOpt Applied Use Case
Metric Management: a SigOpt Applied Use Case
 
SigOpt for Machine Learning and AI
SigOpt for Machine Learning and AISigOpt for Machine Learning and AI
SigOpt for Machine Learning and AI
 
Using Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning ModelsUsing Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning Models
 
Using Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning ModelsUsing Bayesian Optimization to Tune Machine Learning Models
Using Bayesian Optimization to Tune Machine Learning Models
 
DutchMLSchool. ML for Energy Trading and Automotive Sector
DutchMLSchool. ML for Energy Trading and Automotive SectorDutchMLSchool. ML for Energy Trading and Automotive Sector
DutchMLSchool. ML for Energy Trading and Automotive Sector
 
Scaling & Transforming Stitch Fix's Visibility into What Folks will love
Scaling & Transforming Stitch Fix's Visibility into What Folks will loveScaling & Transforming Stitch Fix's Visibility into What Folks will love
Scaling & Transforming Stitch Fix's Visibility into What Folks will love
 
SigOpt at GTC - Tuning the Untunable
SigOpt at GTC - Tuning the UntunableSigOpt at GTC - Tuning the Untunable
SigOpt at GTC - Tuning the Untunable
 
Experiment Management for the Enterprise
Experiment Management for the EnterpriseExperiment Management for the Enterprise
Experiment Management for the Enterprise
 
Machine Learning/ Data Science: Boosting Predictive Analytics Model Performance
Machine Learning/ Data Science: Boosting Predictive Analytics Model PerformanceMachine Learning/ Data Science: Boosting Predictive Analytics Model Performance
Machine Learning/ Data Science: Boosting Predictive Analytics Model Performance
 
MLConf 2016 SigOpt Talk by Scott Clark
MLConf 2016 SigOpt Talk by Scott ClarkMLConf 2016 SigOpt Talk by Scott Clark
MLConf 2016 SigOpt Talk by Scott Clark
 
Scott Clark, Co-Founder and CEO, SigOpt at MLconf SF 2016
Scott Clark, Co-Founder and CEO, SigOpt at MLconf SF 2016Scott Clark, Co-Founder and CEO, SigOpt at MLconf SF 2016
Scott Clark, Co-Founder and CEO, SigOpt at MLconf SF 2016
 
Failure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature DeliveryFailure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature Delivery
 
AI Planning Workshop overview
AI Planning Workshop overviewAI Planning Workshop overview
AI Planning Workshop overview
 
How to Lower the Cost of Deploying Analytics: An Introduction to the Portable...
How to Lower the Cost of Deploying Analytics: An Introduction to the Portable...How to Lower the Cost of Deploying Analytics: An Introduction to the Portable...
How to Lower the Cost of Deploying Analytics: An Introduction to the Portable...
 
From notebook to production with Amazon Sagemaker
From notebook to production with Amazon SagemakerFrom notebook to production with Amazon Sagemaker
From notebook to production with Amazon Sagemaker
 
Exploiting the Enumeration of All Feature Model Configurations: A New Perspec...
Exploiting the Enumeration of All Feature Model Configurations: A New Perspec...Exploiting the Enumeration of All Feature Model Configurations: A New Perspec...
Exploiting the Enumeration of All Feature Model Configurations: A New Perspec...
 
Visualizing Model Selection with Scikit-Yellowbrick: An Introduction to Devel...
Visualizing Model Selection with Scikit-Yellowbrick: An Introduction to Devel...Visualizing Model Selection with Scikit-Yellowbrick: An Introduction to Devel...
Visualizing Model Selection with Scikit-Yellowbrick: An Introduction to Devel...
 

Mehr von SigOpt

Optimizing BERT and Natural Language Models with SigOpt Experiment Management
Optimizing BERT and Natural Language Models with SigOpt Experiment ManagementOptimizing BERT and Natural Language Models with SigOpt Experiment Management
Optimizing BERT and Natural Language Models with SigOpt Experiment ManagementSigOpt
 
Efficient NLP by Distilling BERT and Multimetric Optimization
Efficient NLP by Distilling BERT and Multimetric OptimizationEfficient NLP by Distilling BERT and Multimetric Optimization
Efficient NLP by Distilling BERT and Multimetric OptimizationSigOpt
 
Detecting COVID-19 Cases with Deep Learning
Detecting COVID-19 Cases with Deep LearningDetecting COVID-19 Cases with Deep Learning
Detecting COVID-19 Cases with Deep LearningSigOpt
 
Tuning Data Augmentation to Boost Model Performance
Tuning Data Augmentation to Boost Model PerformanceTuning Data Augmentation to Boost Model Performance
Tuning Data Augmentation to Boost Model PerformanceSigOpt
 
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...SigOpt
 
Machine Learning Infrastructure
Machine Learning InfrastructureMachine Learning Infrastructure
Machine Learning InfrastructureSigOpt
 
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...SigOpt
 
Lessons for an enterprise approach to modeling at scale
Lessons for an enterprise approach to modeling at scaleLessons for an enterprise approach to modeling at scale
Lessons for an enterprise approach to modeling at scaleSigOpt
 
SigOpt at MLconf - Reducing Operational Barriers to Model Training
SigOpt at MLconf - Reducing Operational Barriers to Model TrainingSigOpt at MLconf - Reducing Operational Barriers to Model Training
SigOpt at MLconf - Reducing Operational Barriers to Model TrainingSigOpt
 
Machine Learning Infrastructure
Machine Learning InfrastructureMachine Learning Infrastructure
Machine Learning InfrastructureSigOpt
 
Tips and techniques for hyperparameter optimization
Tips and techniques for hyperparameter optimizationTips and techniques for hyperparameter optimization
Tips and techniques for hyperparameter optimizationSigOpt
 
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...SigOpt
 
Using Optimal Learning to Tune Deep Learning Pipelines
Using Optimal Learning to Tune Deep Learning PipelinesUsing Optimal Learning to Tune Deep Learning Pipelines
Using Optimal Learning to Tune Deep Learning PipelinesSigOpt
 

Mehr von SigOpt (13)

Optimizing BERT and Natural Language Models with SigOpt Experiment Management
Optimizing BERT and Natural Language Models with SigOpt Experiment ManagementOptimizing BERT and Natural Language Models with SigOpt Experiment Management
Optimizing BERT and Natural Language Models with SigOpt Experiment Management
 
Efficient NLP by Distilling BERT and Multimetric Optimization
Efficient NLP by Distilling BERT and Multimetric OptimizationEfficient NLP by Distilling BERT and Multimetric Optimization
Efficient NLP by Distilling BERT and Multimetric Optimization
 
Detecting COVID-19 Cases with Deep Learning
Detecting COVID-19 Cases with Deep LearningDetecting COVID-19 Cases with Deep Learning
Detecting COVID-19 Cases with Deep Learning
 
Tuning Data Augmentation to Boost Model Performance
Tuning Data Augmentation to Boost Model PerformanceTuning Data Augmentation to Boost Model Performance
Tuning Data Augmentation to Boost Model Performance
 
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...
Interactive Tradeoffs Between Competing Offline Metrics with Bayesian Optimiz...
 
Machine Learning Infrastructure
Machine Learning InfrastructureMachine Learning Infrastructure
Machine Learning Infrastructure
 
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...
SigOpt at Uber Science Symposium - Exploring the spectrum of black-box optimi...
 
Lessons for an enterprise approach to modeling at scale
Lessons for an enterprise approach to modeling at scaleLessons for an enterprise approach to modeling at scale
Lessons for an enterprise approach to modeling at scale
 
SigOpt at MLconf - Reducing Operational Barriers to Model Training
SigOpt at MLconf - Reducing Operational Barriers to Model TrainingSigOpt at MLconf - Reducing Operational Barriers to Model Training
SigOpt at MLconf - Reducing Operational Barriers to Model Training
 
Machine Learning Infrastructure
Machine Learning InfrastructureMachine Learning Infrastructure
Machine Learning Infrastructure
 
Tips and techniques for hyperparameter optimization
Tips and techniques for hyperparameter optimizationTips and techniques for hyperparameter optimization
Tips and techniques for hyperparameter optimization
 
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...
 
Using Optimal Learning to Tune Deep Learning Pipelines
Using Optimal Learning to Tune Deep Learning PipelinesUsing Optimal Learning to Tune Deep Learning Pipelines
Using Optimal Learning to Tune Deep Learning Pipelines
 

Kürzlich hochgeladen

Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptaigil2
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024Becky Burwell
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)Data & Analytics Magazin
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.JasonViviers2
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 

Kürzlich hochgeladen (17)

Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .ppt
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 

Tuning for Systematic Trading: Talk 1

  • 1. SigOpt. Confidential. Talk #1 Intuition behind Bayesian optimization with and without multiple metrics SigOpt Talk Series Tuning for Systematic Trading Tobias Andreasen — Machine Learning Engineer Tuesday, March 24, 2020
  • 2. SigOpt. Confidential. Abstract SigOpt provides an extensive set of advanced features, which help you, the expert, save time while increasing model performance via experimentation. Today, we will start out this talking series by giving an overview of general black box optimization, efficient bayesian optimization and end up extending this to multiple competing metrics.
  • 3. SigOpt. Confidential. Motivation 1. Overview of SigOpt 2. How to solve a black box optimization problem 3. Why you should optimize using multiple competing metrics
  • 5. SigOpt. Confidential. Accelerate and amplify the impact of modelers everywhere
  • 6. SigOpt. Confidential. Experiment Insights Optimization Engine Track, analyze and reproduce any model to improve the productivity of your modeling Enterprise Platform Automate hyperparameter tuning to maximize the performance and impact of your models Standardize experimentation across any combination of library, infrastructure, model or task On-Premise Hybrid/Multi Solution: Experiment, optimize and analyze at scale 6
  • 7. SigOpt. Confidential. SigOpt Features Enterprise Platform Optimization Engine Experiment Insights Reproducibility Intuitive web dashboards Cross-team permissions and collaboration Advanced experiment visualizations Usage insights Parameter importance analysis Multimetric optimization Continuous, categorical, or integer parameters Constraints and failure regions Up to 10k observations, 100 parameters Multitask optimization and high parallelism Conditional parameters Infrastructure agnostic REST API Parallel Resource Scheduler Black-Box Interface Tunes without accessing any data Libraries for Python, Java, R, and MATLAB
  • 8. SigOpt. Confidential. How to solve a black box optimization problem2
  • 9. SigOpt. Confidential. Why black box optimization? SigOpt was designed to empower you, the practitioner, to re-define most machine learning problems as black box optimization problems with the benefit of: • Amplified performance — incremental gains in accuracy or other success metrics • Productivity gains — a consistent platform across tasks that facilitates sharing • Accelerated modeling — early elimination of non-scalable tasks • Compute efficiency — continuous, full utilization of infrastructure SigOpt uses an ensemble of Bayesian and Global Optimization methods to solve these black box optimization problems. 9 Black Box Optimization
  • 10. SigOpt. Confidential. Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters Black Box Optimization
  • 11. SigOpt. Confidential. EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Black Box Optimization Better Results
  • 12. SigOpt. Confidential. Hyperparameter Optimization Model Tuning Grid Search Random Search Bayesian Optimization Training & Tuning Evolutionary Algorithms Deep Learning Architecture Search Hyperparameter Search
  • 13. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
  • 14. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
  • 15. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
  • 16. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
  • 17. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
  • 18. SigOpt. Confidential. A graphical depiction of the iterative process 18 Build a statistical model Sequential Model Based Optimization (SMBO)
  • 19. SigOpt. Confidential. A graphical depiction of the iterative process 19 Build a statistical model Choose the next point to maximize the acquisition function Sequential Model Based Optimization (SMBO)
  • 20. SigOpt. Confidential. A graphical depiction of the iterative process 20 Build a statistical model Build a statistical model Choose the next point to maximize the acquisition function Sequential Model Based Optimization (SMBO)
  • 21. SigOpt. Confidential. A graphical depiction of the iterative process 21 Build a statistical model Build a statistical model Choose the next point to maximize the acquisition function Sequential Model Based Optimization (SMBO) Choose the next point to maximize the acquisition function
  • 22. SigOpt. Confidential. Gaussian processes: a powerful tool for modeling in spatial statistics A standard tool for building statistical models is the Gaussian process [Fasshauer et al, 2015, Fraizer, 2018]. • Assume that function values are jointly normally distributed. • Apply prior beliefs about mean behavior and covariance between observations. • Posterior beliefs about unobserved locations can be computed rather easily. Different prior assumptions produce different statistical models: 22 Complexity #1: Optimizing the Model in SMBO
  • 23. SigOpt. Confidential. Acquisition function: given a model, how should we choose the next point? An acquisition function is a strategy for defining the utility of a future sample, given the current samples, while balancing exploration and exploitation [Shahriari et al, 2016]. Different acquisition functions choose different points (EI, PI, KG, etc.). 23 Complexity #2: Optimizing the SMBO Process Exploration: Learning about the whole function f Exploitation: Further resolving regions where good f values have already been observed
  • 24. SigOpt. Confidential. RESTAPI The SigOpt API handles the complexity Better Results
  • 25. SigOpt Blog Posts: Intuition Behind Bayesian Optimization Some Relevant Blog Posts ● Intuition Behind Covariance Kernels ● Approximation of Data ● Likelihood for Gaussian Processes ● Profile Likelihood vs. Kriging Variance ● Intuition behind Gaussian Processes ● Dealing with Troublesome Metrics Find more blog posts visit: https://sigopt.com/blog/
  • 26. SigOpt. Confidential. Why you should optimize using multiple competing metrics 3
  • 27. SigOpt. Confidential. Why optimize against multiple competing metrics? SigOpt allows the user to specify multiple competing metrics for either optimization or tracking to better align success of the experimentation with business value: • Multiple metrics — The option to defining multiple metrics, which can yield new and interesting results • Insights, metric storage — Insights through tracking of optimized and unoptimized metrics • Thresholds — The ability to define thresholds for success to better guide the optimizer This process gives models that deliver more reliable business outcomes by helping optimally make the tradeoffs inherent in the modeling process and real world applications. 27 Optimizing Multiple Competing Metrics
  • 28. SigOpt. Confidential. Your firewall Training Data AI, ML, DL, Simulation Models Model Evaluation or Backtest Testing Data New Configurations Objective Metric EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters Optimizing Multiple Competing Metrics Better Results
  • 29. SigOpt. Confidential. EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Optimizing Multiple Competing Metrics Multiple Optimized and Unoptimized Metrics Better Results
  • 30. SigOpt. Confidential. Looking for the right Balance Balancing competing metrics to find the Pareto frontier Most problems of practical relevance involve 2 or more competing metrics. • Neural networks — Balancing accuracy and inference time • Materials design — Balancing performance and maintenance cost • Algo trading — Balancing Sharpe Ratio and book size In a situation with Competing Metrics, the set of all efficient points (the Pareto frontier) is the solution. 30 Pareto Frontier Feasible Region
  • 31. SigOpt. Confidential. Balancing competing metrics to find the Pareto frontier As shown before, the goal in multi objective or multi criteria optimization the goal is to find the optimal set of solution across a set of function [Knowles, 2006]. • This is formulated as finding the maximum of the set functions f1 to fn over the same domain x • No single point exist as the solution, but we are actively trying to maximize the size of the efficient frontier, which represent the set of solutions • The solution is found through scalarization methods such as convex combination and epsilon-constraint 31 Finding the Best Tradeoffs
  • 32. SigOpt. Confidential. Optimizing Multiple Competing Metrics: First Approach Intuition: Convex Combination Scalarization Idea: If we can convert the multimetric problem into a scalar problem, we can solve this problem using Bayesian optimization. One possible scalarization is through a convex combination of the objectives. 32
  • 33. SigOpt. Confidential. Balancing competing metrics to find the Pareto frontier with threshold As shown before, the goal in multi objective or multi criteria optimization the goal is to find the optimal set of solution across a set of function [Knowles, 2006]. • This is formulated as finding the maximum of the set functions f1 to fn over the same domain x • No single point exist as the solution, but we are actively trying to maximize the size of the efficient frontier, which represent the set of solutions • The solution is found through constrained scalarization methods such as convex combination and epsilon-constraint • Allow users to change constraints as the search progresses [Letham et al, 2019] 33 Optimizing Multiple Competing Metrics: Deeper Approach
  • 34. SigOpt. Confidential. Rephrase the Entire Problem Constrained Scalarization 1. Model all metrics independently. • Requires no prior beliefs of how metrics interact. • Missing data removed on a per metric basis if unrecorded. 2. Expose the efficient frontier through constrained scalar optimization. • Enforce user constraints when given. • Iterate through sub constraints to better resolve efficient frontier, if desired. • Consider different regions of the frontier when parallelism is possible. 3. Allow users to change constraints as the search progresses. • Allow the problems/goals to evolve as the user’s understanding changes. Constraints give customers more control over the circumstances and more ability to understand our actions. 34 Variation on Expected Improvement [Letham et al, 2019]
  • 35. SigOpt. Confidential. Intuition: Scalarization and Epsilon Constraints 35 Use Epsilon Constraints to Focus Search
  • 36. SigOpt. Confidential. Intuition: Constrained Scalarization and Epsilon Constraints 36 Add Thresholds to Limit Frontier
  • 37. Multimetric Use Case 1 ● Category: Time Series ● Task: Sequence Classification ● Model: CNN ● Data: Diatom Images ● Analysis: Accuracy-Time Tradeoff ● Result: Similar accuracy, 33% the inference time Multimetric Use Case 2 ● Category: NLP ● Task: Sentiment Analysis ● Model: CNN ● Data: Rotten Tomatoes Movie Reviews ● Analysis: Accuracy-Time Tradeoff ● Result: ~2% in accuracy versus 50% of training time Learn more https://devblogs.nvidia.com/sigopt-deep-learning- hyperparameter-optimization/ Use Case: Balancing Speed & Accuracy in Deep Learning
  • 38. SigOpt. Confidential. Tobias Andreasen | tobias@sigopt.com For more information visit: https://sigopt.com/research/ Questions?