These slides correspond to a recording of a live webcast of a demo of Metric Management functionality in SigOpt, keeping model size down while increasing validation accuracy for a road sign image classification problem.

1. SigOpt. Confidential.
SigOpt Webinar:
Metric Management
Thursday, May 28, 2020
Barrett Williams — Product Marketing Lead
Harvey Cheng — Research Engineer

2. SigOpt. Confidential.
Accelerate and amplify the
impact of modelers everywhere

3. Opportunity: Standardize process, boost performance
Notebook & Model Framework
Hardware Environment
Data
Preparation
Experimentation, Training, Evaluation
Model
Productionalization
Validation
Serving
Deploying
Monitoring
Managing
Inference
Online Testing
Transformation
Labeling
Pre-Processing
Pipeline Dev.
Feature Eng.
Feature Stores
On-Premise Hybrid Multi-Cloud
Experimentation & Model Optimization
Insights, Tracking,
Collaboration
Model Search,
Hyperparameter Tuning
Resource Scheduler,
Management
...and more

4. Your firewall
Training
Data
AI, ML, DL,
Simulation Model
Model Evaluation
or Backtest
Testing
Data
New
Configurations
Tracked
Objective
Metrics
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
Your data
and models
stay private
Iterative, automated optimization
Integrates
with any
modeling
stack

5. Context: From optimization system to modeling platform
Early Stopping
Multitask
Multisolution
Conditionals
Constraints
Failure Regions
10,000 Observations
100x Parallel
100 Parameters
Model
Optimization
Algorithms
Model
Optimization
System
Patented ensemble
of Bayesian & global
optimization
algorithms for any
combination of
parameter types
Patented system for
millisecond latency
and asynchronous
parallelization
Advanced
Model
Optimization
Experiment
Insights
Full Exp. History
Best-Seen Trace
Parameter Importance
Parallel Coordinates
Multitask Insights
Metric
Management
Multimetric Optimization
Multimetric Visualization
Metric Strategy
Metric Tracking
Metric Thresholds
Metric Constraints
Training
Insights

6. Challenge: Tough to define and select the right metric
Business contexts create
more than one metric
Optimizing on a single
metric is limiting or
inadequate
Need to understand
tradeoffs between
metrics
Multimetric optimization
generates a Pareto
efficient frontier
Metric constraints
consider additional
business constraints
Metric strategy records
all observable metric
values
Need to apply guardrail
to metrics
Neet to track and
monitor additional
information

7. SigOpt. Confidential.
Metric Management Feature How it Helps You
Metric Storage Allows for later analysis

8. SigOpt. Confidential.
Metric Management Feature How it Helps You
Metric Storage Allows for later analysis
Metric Thresholds Help SigOpt focus on optimizations that meet business needs

9. SigOpt. Confidential.
Metric Management Feature How it Helps You
Metric Storage Allows for later analysis
Metric Thresholds Help SigOpt focus on optimizations that meet business needs
Metric Constraints Set boundaries on what metric outcomes are useful for your end result

10. SigOpt. Confidential.
Metric Management Feature How it Helps You
Metric Storage Allows for later analysis
Metric Thresholds Help SigOpt focus on optimizations that meet business needs
Metric Constraints Set boundaries on what metric outcomes are useful for your end result
Metric Strategy Control which metrics are targeted—and how—via API

11. SigOpt. Confidential.
Metric Management Feature How it Helps You
Metric Storage Allows for later analysis
Metric Thresholds Help SigOpt focus on optimizations that meet business needs
Metric Constraints Set boundaries on what metric outcomes are useful for your end result
Metric Strategy Control which metrics are targeted—and how—via API
Multimetric Optimization
Optimize against two metrics at the same time,
and analyze the tradeoff frontier

12. SigOpt. Confidential.
Metric Management Feature How it Helps You
Metric Storage Allows for later analysis
Metric Thresholds Help SigOpt focus on optimizations that meet business needs
Metric Constraints Set boundaries on what metric outcomes are useful for your end result
Metric Strategy Control which metrics are targeted—and how—via API
Multimetric Optimization
Optimize against two metrics at the same time,
and analyze the tradeoff frontier
Observation/metric failures
Mark failed observation states,
to guide SigOpt away from their regions

13. Use Case: Metric Management for Computer Vision
Road Sign Classification task
● Dataset:
German Traffic Sign Recognition Benchmark
● Modeling framework: Keras
● Model type: CNN

14. Now, on to the demo!
Harvey Cheng
SigOpt Research Engineer

15. Feature Use case
Metric Storage
Tracking auxiliary metrics such as training time
and testing accuracy for later analysis.
Multimetric Optimization
Optimizing validation accuracy of the network and the MAC
operations. Understanding tradeoffs.
Metric Constraints Setting thresholds on the size of the network.
Observation/metric failures
Marking diverged networks as failures
to resolve for further investigation.
Demo Recap

16. “Integrating SigOpt with our modeling platform
empowers our team to more efficiently experiment,
optimize and, ultimately, model at scale.”
Peter Welinder
Research Scientist

17. “We’ve integrated SigOpt’s optimization service and
are now able to get better results faster and cheaper
than any solution we’ve seen before.”
Matt Adereth
Managing Director

18. SigOpt. Confidential.
Questions?The Zoom chat is now open for questions.

19. SigOpt. Confidential.
Check out our
YouTube channel:
See the example yourself:
Find the public
SigOpt experiment here.
Try our solution:
Sign up at
sigopt.com/try-it
today.
Click Here
Upcoming webinars:
● Warm Start Tuning with Prior Beliefs
Thursday, June 4 at 10am PT / 1pm ET
● Detecting COVID-19 Cases
with Deep Learning
Tuesday, June 9 at 10am PT / 1pm ET

20. SigOpt. Confidential.
Thank you!