MAU Vegas 2016 — Lessons Learned From A/B Testing Wins and Losses
•
0 gefällt mir•2,689 views
Mobile Apps Unlocked Vegas 2016 Thursday, May 5 Retention Breakout Session — 3:30pm - 3:50pm By now, we hope you have notepad’s full of ideas and new thoughts. If for some reason you have not hit your new things to try quota, don’t worry, our closing session will take you over the top featuring first-hand learnings from Life360's non-stop product experimentation engine. Andreas Gross, Founding Team / Product @ Life360
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Ähnlich wie MAU Vegas 2016 — Lessons Learned From A/B Testing Wins and Losses
Ähnlich wie MAU Vegas 2016 — Lessons Learned From A/B Testing Wins and Losses (20)
Mehr von Grow.co
Mehr von Grow.co (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (6)
MAU Vegas 2016 — Lessons Learned From A/B Testing Wins and Losses
- 1. Lessons Learned From A/B Testing Wins and Losses
- 2. WHO I AM: GROWTH HACKER ANDY
- 3. Growth Team /groTH tem/ n. A group within the company that develops the product experience in a way that creates more activated users ! Pre Product/Market fit: build the growth engine ! Post Product/Market fit: optimize the engine (make it bigger) ! Any time: generate hacks, non-obvious ways to reach your goals Definition
- 5. GET THE PIECES IN PLACE BEFORE ADDING GAS
- 6. Do things that don’t scale
- 8. Test yourself to product market fit What type of product are our users interested in? How can we reach this users the best way? Pour Gas on it WE HAD TO EXPERIMENT WITH A LOT OF THINGS
- 9. How do you build what users actually want? Child ID Dog TrackerFamily Locator Lost Phone Ads Landing Pages
- 10. THINK OF IT AS DIVIDE & CONQUER
- 13. RISKLOW HIGH Big Swings Optimizations REWARD HIGH Start Ups need step function take bold bets
- 14. Scalable growth through testing Lather, Rinse, Repeat. Continuous Improvement Evaluate Results Test Design Develop Hypothesis Implement Leanings
- 15. Developing hypotheses How to Develop Hypotheses 1. Leverage the team, institutional knowledge, understand important “Themes” 2. Qualitative studies, surveys 3. Quantitative analysis
- 16. Test Design Test Design How to Design Tests Effectively 1. Determine right testing format 2. Optimize test to avoid overinvesting 3. Execute quickly, full instrumentation for analysis
- 17. Evaluate Results How to Evaluate Results 1. Analyze core metrics impact of each variant 2. Ensure all numbers are statistically significant 3. Understand long term impacts and side effects Evaluate Results
- 18. Implement Learnings How to Implement Learnings 1. Understand what caused the change 2. Determine how this learnings applies to product aspects 3. Promote visibility of learnings/takeaways Implement Leanings
- 19. Qualitative Tests Painted Door Tests Feature Tests INVESTMENT COSTLOW HIGH Painted Door Top of Funnel Test Full FeatureMany Some Few Types of tests avoid overinvesting
- 20. Qualitative Tests Purpose Determine interest in product via conversations with your users and potential users Key Test as many ideas as possible but keep in mind that people want to please you Example Emergency Contacts ETA Sharing Carpooling Qualitative Tests
- 21. Feature Tests Purpose Building a functional feature and see how users interact Key Don’t build the full feature, start at the top of the funnel and work your way down Example Emergency Contacts Recommender FUE Add EC Send EC Accept EC Create ACT (Partial) Feature Tests
- 22. Aggressive vs. Passive Tests Passive Tests End Point Aggressive Tests LEARNINGSLOW HIGH
- 23. HOW TO TRUST YOUR RESULTS?
- 24. Test Design - Explicit Control Variant Allocation Design variants to have an additional control group ➔ Ensures control group meets test user criteria ➔ Reduce risk of variant assignment issues
- 25. Test Design - Explicit Control Variant Allocation All Users
- 26. Test Design - Explicit Control Variant Allocation Eligible for experiment Ineligible for experiment
- 27. Test Design - Explicit Control Variant Allocation Eligible but excluded from experiment Ineligible for experiment Included in experiment
- 28. Test Design - Explicit Control Variant Allocation Treatment 1 Eligible but excluded from experiment Ineligible for experiment Control Variant 1 Variant 2
- 29. Test Design - Explicit Control Variant Allocation Always consider keeping long-term holdout groups ➔ Understand potential impacts that may not be seen early in the user lifecycle ➔ Enables measurement of how impacts rollup into product growth
- 30. “A GOOD PLAN, VIOLENTLY EXECUTED TODAY, IS BETTER THAN A PERFECT PLAN EXECUTED NEXT WEEK.” GENERAL GEORGE S PATTON
- 31. HERE IS WHAT I AM UP TO NEXT Andreas@reddit.com