by Scott Clark, Co-founder & CEO, SigOpt

1. BAYESIAN GLOBAL OPTIMIZATION
Using Optimal Learning to Tune Predictive Models
Scott Clark
scott@sigopt.com

2. OUTLINE
1. Why is Tuning Models Hard?
2. Comparison of Tuning Methods
3. Bayesian Global Optimization
4. Deep Learning Examples
5. Evaluating Optimization Strategies

4. Machine Learning is
extremely powerful

5. Machine Learning is
extremely powerful
Tuning Machine Learning systems is
extremely non-intuitive

6. https://www.quora.com/What-is-the-most-important-unresolved-problem-in-machine-learning-3
What is the most important unresolved
problem in machine learning?
“...we still don't really know why some configurations of
deep neural networks work in some case and not others,
let alone having a more or less automatic approach to
determining the architectures and the
hyperparameters.”
Xavier Amatriain, VP Engineering at Quora
(former Director of Research at Netflix)

7. Photo: Joe Ross

8. TUNABLE PARAMETERS IN DEEP LEARNING

9. TUNABLE PARAMETERS IN DEEP LEARNING

10. Photo: Tammy Strobel

11. STANDARD METHODS
FOR HYPERPARAMETER SEARCH

12. STANDARD TUNING METHODS
Parameter
Configuration
?
Grid Search Random Search
Manual Search
- Weights
- Thresholds
- Window sizes
- Transformations
ML / AI
Model
Testing
Data
Cross
Validation
Training
Data

13. OPTIMIZATION FEEDBACK LOOP
Objective Metric
Better
Results
REST API
New configurations
ML / AI
Model
Testing
Data
Cross
Validation
Training
Data

14. BAYESIAN GLOBAL OPTIMIZATION

15. … the challenge of how to collect information as efficiently
as possible, primarily for settings where collecting information
is time consuming and expensive.
Prof. Warren Powell - Princeton
What is the most efficient way to collect information?
Prof. Peter Frazier - Cornell
How do we make the most money, as fast as possible?
Scott Clark - CEO, SigOpt
OPTIMAL LEARNING

16. ● Optimize objective function
○ Loss, Accuracy, Likelihood
● Given parameters
○ Hyperparameters, feature/architecture params
● Find the best hyperparameters
○ Sample function as few times as possible
○ Training on big data is expensive
BAYESIAN GLOBAL OPTIMIZATION

17. SMBO
Sequential Model-Based Optimization
HOW DOES IT WORK?

18. 1. Build Gaussian Process (GP) with points
sampled so far
2. Optimize the fit of the GP (covariance
hyperparameters)
3. Find the point(s) of highest Expected
Improvement within parameter domain
4. Return optimal next best point(s) to sample
GP/EI SMBO

19. GAUSSIAN PROCESSES

20. GAUSSIAN PROCESSES

21. GAUSSIAN PROCESSES

22. GAUSSIAN PROCESSES

23. GAUSSIAN PROCESSES

24. GAUSSIAN PROCESSES

25. GAUSSIAN PROCESSES

26. GAUSSIAN PROCESSES

27. overfit good fit underfit
GAUSSIAN PROCESSES

28. EXPECTED IMPROVEMENT

29. EXPECTED IMPROVEMENT

30. EXPECTED IMPROVEMENT

31. EXPECTED IMPROVEMENT

32. EXPECTED IMPROVEMENT

33. EXPECTED IMPROVEMENT

34. DEEP LEARNING EXAMPLES

35. ● Classify movie reviews
using a CNN in MXNet
SIGOPT + MXNET

36. TEXT CLASSIFICATION PIPELINE
ML / AI
Model
(MXNet)
Testing
Text
Validation
Accuracy
Better
Results
REST API
Hyperparameter
Configurations
and
Feature
Transformations
Training
Text

37. TUNABLE PARAMETERS IN DEEP LEARNING

38. ● Comparison of several RMSProp SGD parametrizations
STOCHASTIC GRADIENT DESCENT

39. ARCHITECTURE PARAMETERS

40. Grid Search Random Search
?
TUNING METHODS

41. MULTIPLICATIVE TUNING SPEED UP

42. SPEED UP #1: CPU -> GPU

43. SPEED UP #2: RANDOM/GRID -> SIGOPT

44. CONSISTENTLY BETTER AND FASTER

45. ● Classify house numbers in
an image dataset (SVHN)
SIGOPT + TENSORFLOW

46. COMPUTER VISION PIPELINE
ML / AI
Model
(Tensorflow)
Testing
Images
Cross
Validation
Accuracy
Better
Results
REST API
Hyperparameter
Configurations
and
Feature
Transformations
Training
Images

47. METRIC OPTIMIZATION

48. ● All convolutional neural network
● Multiple convolutional and dropout layers
● Hyperparameter optimization mixture of
domain expertise and grid search (brute force)
SIGOPT + NEON
http://arxiv.org/pdf/1412.6806.pdf

49. COMPARATIVE PERFORMANCE
● Expert baseline: 0.8995
○ (using neon)
● SigOpt best: 0.9011
○ 1.6% reduction in
error rate
○ No expert time
wasted in tuning

50. SIGOPT + NEON
http://arxiv.org/pdf/1512.03385v1.pdf
● Explicitly reformulate the layers as learning residual
functions with reference to the layer inputs, instead of learning unreferenced functions
● Variable depth
● Hyperparameter optimization mixture of domain expertise and grid search (brute force)

51. COMPARATIVE PERFORMANCE
Standard Method
● Expert baseline: 0.9339
○ (from paper)
● SigOpt best: 0.9436
○ 15% relative error
rate reduction
○ No expert time
wasted in tuning

52. EVALUATING THE OPTIMIZER

53. OUTLINE
● Metric Definitions
● Benchmark Suite
● Eval Infrastructure
● Visualization Tool
● Baseline Comparisons

54. What is the best value found after optimization completes?
METRIC: BEST FOUND
BLUE RED
BEST_FOUND 0.7225 0.8949

55. How quickly is optimum found? (area under curve)
METRIC: AUC
BLUE RED
BEST_FOUND 0.9439 0.9435
AUC 0.8299 0.9358

56. STOCHASTIC OPTIMIZATION

57. ● Optimization functions from literature
● ML datasets: LIBSVM, Deep Learning, etc
BENCHMARK SUITE
TEST FUNCTION TYPE COUNT
Continuous Params 184
Noisy Observations 188
Parallel Observations 45
Integer Params 34
Categorical Params / ML 47
Failure Observations 30
TOTAL 489

58. ● On-demand cluster in
AWS for parallel eval
function optimization
● Full eval consists of
~20000 optimizations,
taking ~30 min
INFRASTRUCTURE

59. RANKING OPTIMIZERS
1. Mann-Whitney U tests
using BEST_FOUND
2. Tied results then partially
ranked using AUC
3. Any remaining ties, stay
as ties for final ranking

60. RANKING AGGREGATION
● Aggregate partial rankings across all eval functions using
Borda count (sum of methods ranked lower)

61. SHORT RESULTS SUMMARY

62. BASELINE COMPARISONS

63. SIGOPT SERVICE

64. OPTIMIZATION FEEDBACK LOOP
Objective Metric
Better
Results
REST API
New configurations
ML / AI
Model
Testing
Data
Cross
Validation
Training
Data

65. SIMPLIFIED OPTIMIZATION
Client Libraries
● Python
● Java
● R
● Matlab
● And more...
Framework Integrations
● TensorFlow
● scikit-learn
● xgboost
● Keras
● Neon
● And more...
Live Demo

66. DISTRIBUTED TRAINING
● SigOpt serves as a distributed
scheduler for training models
across workers
● Workers access the SigOpt API
for the latest parameters to
try for each model
● Enables easy distributed
training of non-distributed
algorithms across any number
of models

67. https://sigopt.com/getstarted
Try it yourself!

68. Questions?
contact@sigopt.com
https://sigopt.com
@SigOpt