Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2xsST7Y. Matt Adereth talks about the Black-box optimization techniques, what’s actually going on inside of these black-boxes and discusses an idea of how they can be used to solve problems today. He deep dives into a few of the most popular ones, such as Distributed Nelder-Mead and Bayesian Optimization, and discusses their trade-offs. Filmed at qconnewyork.com. Matt Adereth is a Managing Director at Two Sigma Investments, where he works on tools, infrastructure and methodologies for quantitative financial research. He previously worked at Microsoft on Office, focusing on data connectivity and visualization features. In his spare time, he designs open-source ergonomic keyboards using Clojure.

1. www.twosigma.com
Modern Distributed Optimization
October 6, 2017
Matt Adereth
QCon NY 2017

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
black-box-optimization

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Important Legal Information
The views expressed herein are not necessarily the views of Two Sigma Investments, LP or
any of its affiliates (collectively, “Two Sigma”). The information presented herein is only for
informational and educational purposes and is not an offer to sell or the solicitation of an
offer to buy any securities or other instruments. Additionally, the information is not intended
to provide, and should not be relied upon for investment, accounting, legal or tax advice.
Two Sigma makes no representations, express or implied, regarding the accuracy or
completeness of this information, and you accept all risks in relying on the above
information for any purpose whatsoever

5. • Unknown Function
• Multiple Parameters
• Expensive

7. Agenda
• Real World Problems
• Distributed Algorithm Deep Dive
• How to Apply Now

8. Real World Problems

9. Cluster Configuration
• Number of VMs
• CPU count
• CPU speed per core
• RAM per core
• Disk count
• Disk speed
• Network capacity of the VM

10. Cluster Configuration with CherryPick

11. JVM Tuning

12. JVM Tuning
$ java -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version | wc -l
java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)
823

14. JVM Tuning at Twitter

15. A/B Testing at Yelp

16. Deep Learning

17. Deep Learning
http://playground.tensorflow.org/

18. Algorithms

19. 💻(🎛)
👩(🎛)
🔮(🎛)
• Guess and Check
• Mental Model
• Grid Search
• Coordinate Descent

20. d = 2
n = 3
d = 1
n = 2
d = 3
n = 4

21. © User:Nicoguaro/ Wikimedia Commons / CC-BY-4.0

23. © User:Nicoguaro/ Wikimedia Commons / CC-BY-4.0

24. Multi-Start Nelder-Mead

26. Parallelized Nelder-Mead Options
Speculative
• Single large batch
• All results influence next batch
Multi-start
• Multiple asynchronous tasks
• Local results only influence
local next steps

27. Bayesian Optimization
1. Estimate the underlying function
2. Evaluate the point that is most likely to be the optimum of
underlying function
3. Update the estimate

28. Bayesian Optimization Demo

29. Parallel Bayesian Optimization
Select the set of points that maximize the
likelihood of any of them being the new optimum
Unique benefit: Asynchronous with information sharing!
Parallel Bayesian Global Optimization of Expensive Functions
Jialei Wang, Scott C. Clark, Eric Liu, Peter I. Frazier
arXiv:1602.05149 [stat.ML]

30. Advanced Parallel BO: Freeze-Thaw
What if you can approximate the objective function early?
Freeze-Thaw Bayesian Optimization
Kevin Swersky, Jasper Snoek, Ryan Prescott Adams
arXiv:1406.3896 [stat.ML]

31. How to Apply Now

32. How to apply now?
• Packages
• Spearmint
• GPyOpt
• BOAT
• Generalized Services
• MOE
• SigOpt
• Specialized Services
• Skipjaq

33. Thanks!
@adereth

34. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
black-box-optimization