SigOpt's Fay Kallel, Head of Product, and Jim Blomo, Head of Engineering, describe the latest updates to SigOpt, a suite of features that help you manage your modeling process.
3. SigOpt. ConïŹdential.
Productivity
75%
of enterprises build their
own AI models
Sources:
1) 75% of advanced enterprises develop their own models - OâReilly Strata State of ML Survey
2) AI projects take 6 months and 2 in 3 of them fail - Databricks 2019 CIO Survey
3) 58% will put AI in prod by 2021 - Gartner 2019 CIO Survey | Gartner: Predicts 2020: ArtiïŹcial Intelligence â the Road to Production
Performance
2 of 3
modeling projects fail to be
productionalized
Scale
58%
of teams will deploy 20+
models in production
4. âCompanies lose patience and fail before they go to production
because they have no ability to understand where the real
issues are in the life cycle.â
Global Online Retailer
5. âML organizations and engineering biggest problem is the
art of ïŹnding how oïŹine metrics may fail and therefore what
can we do to better design it so the learning system can
perform and achieve our business metrics.â
Robotics Startup
6. âWith DL and neural networks, HPO is absolutely necessary,
sometimes swapping a grid search for HPO is suïŹcient to drive
a 1% improvement and that is a game changer in some
problems.â
Cloud Services Provider
7. âWe hired experts from Google, Amazon, Facebook, Microsoft and they
have diverse skills across tools, we canât aïŹord to standardize on one
stack so we need to make them productive by giving them the tools to
manage all of their models regardless of their modeling environment,
and that challenge took us a few years to begin solvingâ
Global Hospitality Services Provider
8. âwe have decentralized organization and as a result, we have
many teams building models, leveraging each otherâs work is
crucial yet very hard to do today.â
Global Enterprise Solution Provider
10. SigOpt. ConïŹdential.
SigOpt enabling MLOps in the Enterprise
Prepare
Data
Data
Repository
Train
Model
FEATURIZE TRAIN EVALUATE TUNE
Model
Registry
MODELING PIPELINE Data
Scientist
ML Engineer
/ DevOps
Release
Model
APPROVE VALIDATEDEPLOY
MONITOR
DEPLOYMENT PIPELINE
COLLECTData
Source
Data
Engineer
12. BeneïŹts: Explore, Understand, Advance
Realized by global consulting ïŹrm
with 3,000+ modelers
30% Productivity Boost
Realized by Two Sigma in
comparison to open source
8x Faster Experimentation
Custom visualization informs
intuition and explains progress
Interactively visualize learning
Seamless training to tuning to
reaching the best model
Train & Tune Any Model
Retrace and reproduce modeling
decisions with perfect recall
Transcribe modeling workïŹow
14. SigOpt. ConïŹdential.
How it Works: Integrate with a few lines of code
Install SigOpt1
Instrument model2
Track runs3
Run optimization loop4
Analyze experiments5
Install SigOpt1
15. SigOpt. ConïŹdential.
How it Works: Integrate with a few lines of code
Install SigOpt1
Instrument model2
Track runs3
Run optimization loop4
Analyze experiments5
Install SigOpt1
16. SigOpt. ConïŹdential.
How it Works: Integrate with a few lines of code
Install SigOpt1
Instrument model2
Track runs3
Run optimization loop4
Analyze experiments5
Install SigOpt1
17. SigOpt. ConïŹdential.
How it Works: Integrate with a few lines of code
Install SigOpt1
Instrument model2
Track runs3
Run optimization loop4
Analyze experiments5
Install SigOpt1
18. SigOpt. ConïŹdential.
How it Works: Analyzing Results
Install SigOpt1
Instrument model2
Track runs3
Run optimization loop4
Analyze experiments5
Install SigOpt1
19. SigOpt. ConïŹdential.
How it Works: Analyzing Results
Install SigOpt1
Instrument model2
Track runs3
Run optimization loop4
Analyze experiments5
Install SigOpt1
21. âThank you so much for your work on this, the cool factor is high, the
fact that you are hovering over a metric point and showing me how
that point is linked to all metrics and hyperparameters, that dashboard
is giving me in one view great intuition across two types of metrics, I
commend you for coming up with an interface like this.â
Beta tester from a Financial Insurance Provider
23. âSaved views after ïŹltering is awesome! It solved my
current workaround where I write a script to pull from
hundreds of directories.â
Beta tester from the U.S. Federal Government
25. SigOpt. ConïŹdential.
ML, DL or
Simulation Model
Model Evaluation or
Backtest
Testing
Data
Training
Data
Never
accesses your
data or models
Iterative, automated optimization
Built specifically
for scalable
enterprise use
cases
25
What is âOptimizeâ?
REST API
28. SigOpt. ConïŹdential.
â API Performance Suggestions and observations are produced and recorded in
milliseconds.
â 100+ Parameters per Model Support for your most complex models, with
continuous, integer, and categorical types.
10,000+ Observations per Experiment Run thousands of simulations with no
slowdown of the API.
â 1,000,000+ Experiments Optimize and re-tune every model in production to
ensure continuous optimal performance.
â 100s of Simultaneous Experiments & Observations With our Cloud solution, high
concurrent workload is automatically scaled and supported.
â 1000s of modelers Enable your entire modeling organization.
Enterprise Platform: Scalability
28