Modeling at Scale: SigOpt at TWIMLcon 2019
•
1 gefällt mir•813 views
The document discusses how to accelerate and amplify the impact of modelers. It describes SigOpt's platform which allows for automated hyperparameter optimization, tracking of experiments, and reuse of insights. This helps make modeling faster, cheaper, and better. The document advocates balancing flexibility and standardization, maximizing resource utilization through techniques like parallelization, and unlocking new capabilities such as optimizing expensive models or exploring architectures.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (18)
Ähnlich wie Modeling at Scale: SigOpt at TWIMLcon 2019
Ähnlich wie Modeling at Scale: SigOpt at TWIMLcon 2019 (20)
Mehr von SigOpt
Mehr von SigOpt (19)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Modeling at Scale: SigOpt at TWIMLcon 2019
- 2. Accelerate and amplify the impact of modelers everywhere
- 3. SigOpt. Confidential.3 How I got here: 10+ years of tuning models
- 4. Differentiated Models Tailored Models 10,000x Analytics 2.0 Models 100x 1x Modelers by Segment Value per Model Enterprise AI Goals: Differentiate Products Generate Revenue Requirements: Modelers with Expertise Best-in-Class Solutions
- 5. SigOpt. Confidential. Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters Your data and models stay private Iterative, automated optimization Integrates with any modeling stack
- 6. $300B+ in assets under management Current SigOpt algorithmic trading customers represent $500B+ in market capitalization Current SigOpt enterprise customers across six industries represent
- 7. SigOpt. Confidential. “We’ve integrated SigOpt’s optimization service and are now able to get better results faster and cheaper than any solution we’ve seen before.” Matt Adereth Managing Director
- 8. Lessons 1. Balance flexibility with standardization 2. Maximize resource utilization 3. Unlock new modeling capabilities
- 9. Balance flexibility with standardization1
- 10. Simulations & Evaluation Modeling FrameworkData Management Data access Data pipelines Data labeling Feature repo. Feature prov. Backfills, versioning Compute Environment Model Execution Model Serving Instrumentation Data validation Phased deploys Online A/B tests Batch scoring On-Premise Optimization & Experimentation Hybrid Multi-Cloud Tracking, Visualization, Version Control Automated Hyperparameter Optimization Distributed Tuning & Job Scheduling Workflows Auto Feature & AutoML Coding Environment Framework Support Library Backtests Metric Iteration Model Evaluation Portfolio Optimization
- 11. Simulations & Evaluation Strategy: Customize and differentiate Modeling Framework Strategy: Modeler choice and flexibility Data Management Strategy: Customize and differentiate Compute Environment Strategy: Modeler choice and flexibility Model Execution Strategy: Customize and differentiate Optimization & Experimentation Strategy: Standardize and scale
- 12. Simulations & Evaluation Strategy: Customize and differentiate Modeling Framework Strategy: Modeler choice and flexibility Data Management Strategy: Customize and differentiate Compute Environment Strategy: Modeler choice and flexibility Model Execution Strategy: Customize and differentiateOptimization & Experimentation Strategy: Standardize and scale
- 13. Simulations & Evaluation Strategy: Customize and differentiate Modeling Framework Strategy: Modeler choice and flexibility Data Management Strategy: Customize and differentiate Compute Environment Strategy: Modeler choice and flexibility Model Execution Strategy: Customize and differentiate Optimization & Experimentation Strategy: Standardize and scale
- 14. Simulations & Evaluation Strategy: Customize and differentiate Modeling Framework Strategy: Modeler choice and flexibility Data Management Strategy: Customize and differentiate Compute Environment Strategy: Modeler choice and flexibility Model Execution Strategy: Customize and differentiate Optimization & Experimentation Strategy: Standardize and scale
- 15. Data Management Data access Data pipelines Data labeling Feature repo. Feature prov. Backfills, versioning Simulation & Evaluation Modeling Framework Hardware Environment On-Premise Hybrid Multi-Cloud Optimization & Experimentation Insights, Tracking, Collaboration Model Search, Hyperparameter Tuning Resource Scheduler, Management Backtests Metric Iteration Model Evaluation Portfolio Optimization Model Execution Model Serving Instrumentation Data validation Phased deploys Online A/B tests Batch scoring Flexibility + Standardization
- 16. SigOpt. Confidential. Benefits of Robust Experimentation & Optimization Learn fast, fail fast Give yourself the best chance at finding good use cases while avoiding false negatives Connect outputs to outcomes Define, select and iterate on your metrics with end-to-end evaluation Find the global maximum Early non-optimized decisions in the process limit your ability to maximize performance Boost productivity Automate modeling tasks so modelers spend more time applying their expertise
- 20. SigOpt. Confidential. Hyperparameter Optimization Model Tuning Grid Search Random Search Bayesian Optimization Training & Tuning Evolutionary Algorithms Deep Learning Architecture Search Hyperparameter Search
- 21. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
- 22. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
- 23. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
- 24. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
- 25. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
- 26. Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 100x Asynchronous Parallelization
- 27. SigOpt. Confidential. 24 Days to optimize a model without SigOpt 3 Days to optimize a model with SigOpt Learn more: https://twosigma.com/news/article/why-two-sigma-is-using-sigopt-for-automated-parameter-tuning/
- 28. Unlock new modeling capabilities3
- 30. SigOpt. Confidential. Unlock new modeling capabilities 30 Accelerate Model Optimization ● Multitask optimization efficiently optimizes “expensive” models ● Relevant docs ● Image classification use case Solve for Competing Business Objectives ● Multimetric optimization empowers you to evaluate trade-offs ● Relevant docs ● Sequence classification use case Explore Model Architectures ● Conditional parameters result in more efficient architecture search ● Relevant docs ● NLP use case
- 31. SigOpt. Confidential. Train and tune expensive models 31 Accelerate Model Optimization ● Multitask optimization efficiently optimizes “expensive” models ● Relevant docs ● Image classification use case Solve for Competing Business Objectives ● Multimetric optimization empowers you to evaluate trade-offs ● Relevant docs ● Sequence classification use case Explore Model Architectures ● Conditional parameters result in more efficient architecture search ● Relevant docs ● NLP use case Combine the intelligence of Bayesian optimization with the efficiency of early termination techniques
- 32. SigOpt. Confidential. Compare and select the right metric 32 Accelerate Model Optimization ● Multitask optimization efficiently optimizes “expensive” models ● Relevant docs ● Image classification use case Solve for Competing Business Objectives ● Multimetric optimization empowers you to evaluate trade-offs ● Relevant docs ● Sequence classification use case Explore Model Architectures ● Conditional parameters result in more efficient architecture search ● Relevant docs ● NLP use case Optimize multiple metrics at the same time to inform your metric definition and selection process
- 33. SigOpt. Confidential. Identify the right tool for the job 33 Accelerate Model Optimization ● Multitask optimization efficiently optimizes “expensive” models ● Relevant docs ● Image classification use case Solve for Competing Business Objectives ● Multimetric optimization empowers you to evaluate trade-offs ● Relevant docs ● Sequence classification use case Explore Model Architectures ● Conditional parameters result in more efficient architecture search ● Relevant docs ● NLP use case Take into account the conditionality of certain parameter types in the optimization process
- 34. Lessons 1. Balance flexibility with standardization 2. Maximize resource utilization 3. Unlock new modeling capabilities
- 35. Try our solution Our team is here to help. Find our table at Booth 02, and sign up to demo SigOpt today. Join our webinar on modeling platforms https://sigopt.com/company /events/webinar-capabilities -ml-platforms/ Listen to our recent podcast with Two Sigma https://twimlai.com/twiml-talk-273-supporting -rapid-model-development-at-two-sigma- with-matt-adereth-scott-clark/ Download TWIML’s Modeling Platforms ebook https://sigopt.com/guide-for- machine-learning-platforms/