N=10^9: Automated Experimentation at Scale
•
3 gefällt mir•1,295 views
Wojciech Galuba, Decision Tools Lead, Facebook Experimentation is a valuable tool for supporting product decisions, iterating on features and gaining actionable insights into people's behavior. In this session Wojciech Galbua, Data Scientist at Facebook, presents an overview of Facebook's experimentation framework and how it is used to make day-to-day data-driven decisions at global scale. The talk focuses on the challenges of building and scaling the analytics infrastructure, designing the tools for ease-of-use and ensuring broad adoption of sound experimentation methodologies across all the teams.
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (13)
Andere mochten auch
Ähnlich wie N=10^9: Automated Experimentation at Scale
Ähnlich wie N=10^9: Automated Experimentation at Scale (20)
Mehr von Optimizely
Mehr von Optimizely (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
N=10^9: Automated Experimentation at Scale
- 1. N=109 Automated Experimenta5on at Scale Wojciech Galuba Decision Tools Lead, Facebook @wgaluba
- 3. N=109: Automated Experimentation at Scale Wojtek Galuba (wgaluba@fb) Decision Tools Team Lead Data Science Infrastructure Facebook
- 4. History of Data Science Infra at FB • Founded April 2012 • A group of data scientists and software engineers • Experienced first hand the need for better infrastructure • Need continues to grow • Team doubled over the past year • Expect continued rapid growth this year
- 5. Why do we experiment?
- 7. Experiment to: Catch problems before they arise
- 8. Experiment to: Choose between multiple options
- 9. Experiment to: Challenge intuitions about product
- 10. Experiment to: Not only evaluate ideas but generate new ones
- 11. Challenges
- 12. Many experiments • Experiments running in parallel • Modifying many different aspects of the product • Overlaps are possible and may conflict
- 13. Many metric dimensions • Different contexts of user actions • Thousands of device types • Geography • Demographics • Time • Enormous space of possible questions
- 14. Many teams • Many ways to run an experiment • Diverse audience for results • Huge set of results from every experiment • Many ways to interpret results
- 16. An experiment
- 17. QuickExperiment Dividepeoplerandomly color: blue size: medium" color: blue" size: big" color: green" size: medium"
- 18. QuickExperiment • Centralized experiment management • Purely config-level: no code pushes to iterate • Automatic exposure logging
- 19. PlanOut
- 20. PlanOut • Open sourced: http://facebook.github.io/planout/ • Flexible experimental design • Full, programmatic control over param values
- 21. Experiment evaluation Exposures Metrics % change from control to test -1 0 1 2 -2 3 -3 posts 99.9 %99 %95 %Confidence:
- 22. Assess decision risk 99.9 %99 %95 %Confidence:
- 23. Lessons learned
- 24. Computing answers to exponential number of possible questions Pre-compute • low specificity • low dimensionality • long-term Compute on-the- fly • high specificity • high dimensionality • short-term A balancing act
- 25. Tackling many dimensions Two sets of tools For exploration For extraction
- 27. Enforce a lifecycle; In particular: clear experiment end dates
- 28. Why lifecycle policy? • Unifies methodology across teams • Prevents tech debt buildup • Minimizes bad impact on product
- 29. Ease of rapid iteration; Safe and scientifically valid iteration
- 30. Fast, but not too fast • Novelty effect vs. top engaged users bump • Understand if waiting helps
- 31. Ensure mutual exclusion; Across platforms, features and infra
- 32. Why mutual exclusion? • Fewer experiment conflicts • Lower metrics variance
- 33. Exposure log everything • Measure effects on the exposed only • Conditioning analyses on the time since last exposure
- 34. The culture Experimentation gives focus; But watch out for tunnel vision!
- 35. The culture Cultivate sound practices; Safe and low-impact experimentation
- 36. The culture Educate on data interpretation; Uniform decision-making across teams
- 37. Understanding uncertainty “Robust misinterpretation of confidence intervals” Rink Hoekstra et al. Psychonomic Bulletin & Review • Only 3% of scientists got all 6 answers right... • How do we educate the users of the tools?
- 38. The three stages of experimentation infrastructure
- 39. Stage 1: Artisanal Photo credit: Abhisek
- 40. Stage 2: Power tools
- 41. Stage 2: Power tools
- 42. Stage 3: Industrialized Photo credit: Steve Jurvetson
- 43. Conclusions Empower, but don’t overwhelm
- 44. Conclusions Filter and automate, but maintain broad focus
- 45. Conclusions Clean data and powerful tools are great, but building the right experimentation culture is equally important
- 46. N=109 Automated Experimenta5on at Scale Wojciech Galuba Decision Tools Lead, Facebook @wgaluba