Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Product Experimentation Pitfalls & How to Avoid Them

138 Aufrufe

Veröffentlicht am

More and more teams are looking for ways to iterate gradually and validate their ideas with data. Techniques like A/B testing, feature flagging, and gradual rollouts are quickly going from niche to mainstream. More experimentation means faster development and better products.

But like any trend, product experimentation is a good idea that can easily go wrong. For every game-changing A/B test, there’s a trail of testing mistakes that led well-meaning teams down the wrong path.

In this webinar, Jon Noronha, Director of Product Management at Optimizely, shares some common ways he’s seen A/B testing go wrong, along with tips for avoiding these pitfalls.

What you’ll learn:

- The most common mistakes product teams make when running experiments
- How to scale experimentation across multiple teams and squads
- How the world’s top technology companies are able to experiment on all product decisions

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Product Experimentation Pitfalls & How to Avoid Them

  1. 1. J O N N O R O N H A D I R E C T O R , P R O D U C T M A N A G E M E N T Building a Culture of Experimentation Avoiding the Pitfalls T W I T T E R : @ T H A T S J O N S E N S E E M A I L : J O N @ O P T I M I Z E L Y . C O M
  2. 2. LAUNCH BUILD DESIGN PRAY EXPERIMENT ITERATE
  3. 3. LAUNCH BUILDDESIGN PRAY EXPERIMENT ITERATE
  4. 4. LAUNCHBUILDDESIGN PRAY EXPERIMENT ITERATE
  5. 5. LAUNCHBUILDDESIGN PRAY EXPERIMENT ITERATE
  6. 6. LAUNCHBUILDDESIGN PRAY EXPERIMENT ITERATE
  7. 7. DESIGN BUILD EXPERIMENT ITERATE LAUNCH PRAY
  8. 8. Product Experimentation Think hypothetically
 • Painted doors • Validating MVPs • Qualitative research • Exploratory A/B testing Mitigate development risk • Frequent, small releases • Feature flagging • Staged rollouts • Safe rollbacks Quantify product impact • Feature validation • Multivariate tests • Iterative adjustments • Measuring business impact Experimentation
  9. 9. Old Reality Culture of Experimentation Top-Down Innovation Embrace Success Make Decisions Follow Orders Bottom-Up Innovation Embrace Failure Validate Decisions Follow Data
  10. 10. A/B TESTING CULTURE OF EXPERIMENTATION
  11. 11. CULTURE OF EXPERIMENTATION CULTURE OF EXPERIMENTATION
  12. 12. 0 10 100 1000 10000 VELOCITY M AT U R I T Y EXPERIMENTATION HERO
  13. 13. 0 10 100 1000 10000 VELOCITY M AT U R I T Y EXPERIMENTATION PROGRAM EXPERIMENTATION HERO
  14. 14. 0 10 100 1000 10000 VELOCITY M AT U R I T Y EXPERIMENTATION PROGRAM CULTURE OF EXPERIMENTATION EXPERIMENTATION HERO
  15. 15. 0 10 100 1000 10000 VELOCITY M AT U R I T Y EXPERIMENTATION HERO EXPERIMENTATION PROGRAM CULTURE OF EXPERIMENTATION
  16. 16. The World’s Largest Digital Laboratory 1,000,000+ Experiments and Counting
  17. 17. 0 10 100 1000 10000 VELOCITY M AT U R I T Y EXPERIMENTATION HERO EXPERIMENTATION PROGRAM CULTURE OF EXPERIMENTATION
  18. 18. Experimentation Pitfalls #1 Optimizing the wrong metrics
  19. 19. Bookings per Session Repeat Bookings per Visitor
  20. 20. Queries per Unique User
  21. 21. clicks that trigger a new query clicks that don’t count as a new search
  22. 22. “We’re the only website on earth that tries to get rid of our users as quickly as possible”
  23. 23. Queries per Unique User Queries per Session Sessions per User
  24. 24. Tips for choosing the right metrics Put yourself in the user’s shoes Ask: what if this went up and nothing else? Constantly re-evaluate Trust your gut!
  25. 25. Experimentation Pitfalls #2 Getting tricked by statistics
  26. 26. Variations A B C D Metrics 1 2 3 4 5
  27. 27. Variations A B C D Metrics 1 2 3 4 5
  28. 28. Variations A B C D Metrics 1 2 3 4 5
  29. 29. Be careful with peeking!
  30. 30. p-Value < 5%. Significant! p-Value > 5%. Inconclusive. p-Value > 5%. Inconclusive. Min Sample Size Time Experiment Starts p-Value > 5%. Inconclusive.
  31. 31. Why is this a problem? There is a ~5% chance of seeing a false positive each time you peek.
  32. 32. p-Value < 5%. Significant! p-Value > 5%. Inconclusive. p-Value > 5%. Inconclusive. Min Sample Size Time Experiment Starts p-Value > 5%. Inconclusive. 4 peeks —> ~18% chance of seeing a false positive
  33. 33. © Randall Patrick Munroe, xkcd.com Beware of multiple comparisons
  34. 34. © Randall Patrick Munroe, xkcd.com
  35. 35. So what can you do? Get a professional! or Use a testing platform that protects you (look for sequential testing and false discovery rate control)
  36. 36. Experimentation Pitfalls #3 Thinking too small
  37. 37. Theproblemwith“A/Btesting”
  38. 38. Theproblemwith“A/Btesting”
  39. 39. 2 Variations A B 3 Variations C 4 Variations D 5 Variations E >6 Variations FA B
  40. 40. Three-fourthsofallexperimentsonlyhave2variations 2 Variations 3 Variations 4 Variations 5 Variations >6 Variations 77% 14% 5% 2% 1%
  41. 41. Testing5ormorevariationscanimproveyourwinrateby75% +71% +75% +48% +32% 25% 33% 37% 44% 43% +75% 2 Variations 3 Variations 4 Variations 5 Variations >6 Variations Significant uplift Significant reduction Inconclusive
  42. 42. “Every day, we run a thousand concurrent experiments to quickly validate new ideas. These experiments run across all our products, from mobile apps and tools used by hoteliers to customer service phone lines and internal systems. Experimentation has become so ingrained in Booking.com culture that every change, from entire redesigns and infrastructure changes to bug fixes, is wrapped in an experiment.”
  43. 43. Experimental Treatment Server-Side Experimentation Optimizely Full Stack Client-Side Experimentation Optimizely Web Client Snippet Experiment Dashboard Default Treatment Server Experimental Treatment Experimental Treatment Client Server SDK Experiment Dashboard User User
  44. 44. Experimentation Pitfalls #4 Hoarding insights
  45. 45. HYPOTHESIS CREATIVE DEVELOPMENT SETUP & QA TESTING ANALYSIS SHARE
  46. 46. Remove all contact buttons for EMEA +1 Pop-Up Promoting the live stream Add NYC + Atlanta User Groups to Log-in 10/17 Navigation Experiment (oct release) AB test EXO handbook LP Test new hero nav on Blog Scroll PopupBlog Posts ? Personalize Promo cards based on behavior August Product Release Butterbar DONE? Test making "try it free" button green Add PhoneTrackingfor Leads Test old homepage vs.new homepage
  47. 47. Atlanta User Groups to Log-in 10/17 gation eriment (oct release) st dbook LP Test new hero nav on Blog Scroll PopupBlog Posts ? Personalize Promo cards based on behavior Test making "try it free" button green Add PhoneTrackingfor Leads
  48. 48. Experimentation Hero 10’s experiments / year Experimentation Program 100’s experiments / year Culture of Experimentation 1000’s experiments / year Scaling the Experimentation Program
  49. 49. 0 10 100 1000 10000 VELOCITY M AT U R I T Y EXPERIMENTATION HERO EXPERIMENTATION PROGRAM CULTURE OF EXPERIMENTATION
  50. 50. Product Experimentation Think hypothetically
 • Painted doors • Validating MVPs • Qualitative research • Exploratory A/B testing Mitigate development risk • Frequent, small releases • Feature flagging • Staged rollouts • Safe rollbacks Quantify product impact • Feature validation • Multivariate tests • Iterative adjustments • Measuring business impact Experimentation
  51. 51. J O N N O R O N H A D I R E C T O R , P R O D U C T M A N A G E M E N T Thank you! Questions? T W I T T E R : @ T H A T S J O N S E N S E E M A I L : J O N @ O P T I M I Z E L Y . C O M

×