Presenter: Anthony Rindone You might have run a few A-B tests, or maybe you've run 100's. But how do you know you're making the right decisions from those tests? Are there questions you aren't considering? How should you analyze the your testing or project portfolio? Are there errors you may be making, but you don't even know it? Senior Product Manager & Storyteller @DataXu. When not overly-caffeinated and talking product strategy frameworks, I am an active yogi, snowboarder, and surfer. Also a cheesecake enthusiast.

1. Making Decisions With Data:
Advanced A/B Testing
Anthony Rindone
Senior Product Manager, Personalization –
Tripadvisor
Twitter: @AnthonyRindone
www.MindfulProductManager.com

2. Session Evaluations

3. We’re Discussing Beyond The Basics…
1-1 Personalization &
Automation
A/B & Multivariate Testing
“On-Demand” or Ad-Hoc
Analytics
Reporting & KPI Scorecards
Raw Data Collected
Analytics Themes – Increasing in Maturity:
Focus for Today

4. Making Decisions with Data: A-B Testing
Secrets They Don’t Want You To Know!
1.
1.
1.
1.
1.

5. Making Decisions with Data: Advanced A/B
Testing Principles
1.
1.
1.
1.
1.
❑ What set of actionable
guidelines can help us make
better
Product/Marketing/Operations
decisions?
❑ What does it mean to make
“better” decisions?

7. Quantifying test groups

8. Principle 1: Averages are your enemy -
focus on time series

9. Principle 1: Averages are your enemy -
focus on time series
Confidence:
Mean:
Test vs. Control:
Sample Size:

10. Principle 1: Averages are your enemy -
focus on time series
16% increase in conversion! We
win!...Right???
Confidence:
Mean:
Test vs. Control:
Sample Size:

11. Principle 1: Averages are your enemy -
focus on time series
16% increase in conversion! We
win!...Right???
Confidence:
Mean:
Test vs. Control:
Sample Size:
Anti-Pattern: Results are outputs as summary tables in Excel with averages leading the way. All
days and all users are considered equal.
Better Solution:

12. Principle 1: Averages are your enemy -
focus on time series
Better Solution:
Expose hidden
trends via cohort
views and time
series
Look for “steady-
state” or equilibrium
trends

13. Applying Results

14. Principle 2: Beware of perverse incentives -
focus on the bigger picture (system)

15. Principle 2: Beware of perverse incentives -
focus on the bigger picture
Hypothetical Product Manager Goal: Grow app
downloads by 10%

16. Principle 2: Beware of perverse incentives -
focus on the bigger picture

17. Principle 2: Beware of perverse incentives -
focus on the bigger picture

18. Principle 2: Beware of perverse incentives -
focus on the bigger picture
Anti-Pattern: Picking specific KPI and/or
optimizing in a vacuum (ex: grow app downloads
by 10%)
Better Solution: A single metric that determines
the success and failure of the test that will
determine what’s best for your site as a whole.
- Everything else are drivers or “secondary”
KPI
Examples: 7-day revenue-per-user (7D$RPU)
post-exposure, daily active users (DAU)

19. Confidence (statistical &
otherwise)

20. Principle 3: Everything is a bet - focus on
adaptability

21. Principle 3: Everything is a bet - focus on
adaptability
Problem: How long do we need to run this test for the data to be significant? (Conversion
Rate)

22. Principle 3: Everything is a bet - focus on
adaptability
Problem: How long do we need to run this test for the data to be significant?
● What’s current “conversion rate?

23. Principle 3: Everything is a bet - focus on
adaptability
Problem: How long do we need to run this test for the data to be significant?
● What’s current “conversion rate?
● What’s daily traffic look like?

24. Principle 3: Everything is a bet - focus on
adaptability
Problem: How long do we need to run this test for the data to be significant?
● What’s current “conversion rate?
● What’s daily traffic look like?
● Assumption: “significant” = 95% confidence with binomial distribution.

25. Principle 3: Everything is a bet - focus on
adaptability
Problem: How long do we need to run this test for the data to be significant?
● What’s current “conversion rate?
● What’s daily traffic look like?
● Assumption: “significant” = 95% confidence with binomial distribution.
● Analyst recommends 3 months ...or 2 days (!!)

26. Principle 3: Everything is a bet - focus on
adaptability
Anti-Pattern: How long do we need to run this test for the data to be significant?
● What’s current “conversion rate?
● What’s daily traffic look like?
● Assumption: “significant” = 95% confidence with binomial distribution.
● Analyst recommends 3 months or 2 days (!!)
Better Solution: Balance risk appropriately; separate “significance” from “confidence
in the data.” (Especially pertinent for lower-traffic areas / sites.)

27. Telling a story

28. Principle 4: Beware of post-hoc storytelling - focus on
good a priori hypothesizing

29. Principle 4: Beware of post-hoc storytelling - focus on
good a priori hypothesizing

30. Principle 4: Beware of post-hoc storytelling - focus on
good a priori hypothesizing
Anti-Pattern: Cherry-picking data that proves your story - and tossing out everything that
doesn’t

31. Principle 4: Beware of post-hoc storytelling - focus on
good a priori hypothesizing
Anti-Pattern: Cherry-picking data that proves your story - and tossing out everything that
doesn’t
Better Solution: Focus on your model, your Primary KPI, and your drivers that you are
testing - everything else can be new hypothesis to test later

32. Planning

33. Principle 5: Avoid Product-Centric mentality
- focus on User-Centric approaches

34. Principle 5: Avoid Product-Centric mentality
- focus on User-Centric approaches
Peter Fader - Customer Centricity: Focus on the Right Customers for
Strategic Advantage:
“Not all customers deserve your best efforts: in the world of
customer centricity, there are good customers…and then there is
pretty much everybody else.”
Anti-Pattern Hypotheses:

35. Principle 5: Avoid Product-Centric mentality
- focus on User-Centric approaches
Peter Fader - Customer Centricity: Focus on the Right Customers for
Strategic Advantage:
“Not all customers deserve your best efforts: in the world of
customer centricity, there are good customers…and then there is
pretty much everybody else.”
Anti-Pattern Hypotheses:
● “We’ll launch this feature as long as long as it doesn’t tank the site.” -- (Smoke Tests)

36. Principle 5: Avoid Product-Centric mentality
- focus on User-Centric approaches
Peter Fader - Customer Centricity: Focus on the Right Customers for
Strategic Advantage:
“Not all customers deserve your best efforts: in the world of
customer centricity, there are good customers…and then there is
pretty much everybody else.”
Anti-Pattern Hypotheses:
● “We’ll launch this feature as long as long as it doesn’t tank the site.” -- (Smoke Tests)
○ Better: Not always bad, but keep minimal in your testing portfolio.
○ “We’ll launch this feature as long as users respond at least neutrally according
to X KPI.”

37. Principle 5: Avoid Product-Centric mentality
- focus on User-Centric approaches
Peter Fader - Customer Centricity: Focus on the Right Customers for
Strategic Advantage:
“Not all customers deserve your best efforts: in the world of
customer centricity, there are good customers…and then there is
pretty much everybody else.”
Anti-Pattern Hypotheses:
● “We’ll launch this feature as long as long as it doesn’t tank the site.” -- (Smoke Tests)
○ Better: Not always bad, but keep minimal in your testing portfolio.
○ “We’ll launch this feature as long as users respond at least neutrally according
to X KPI.”
● “We will know we are successful when conversion increases x% with this test.”

38. Principle 5: Avoid Product-Centric mentality
- focus on User-Centric approaches
Peter Fader - Customer Centricity: Focus on the Right Customers for
Strategic Advantage:
“Not all customers deserve your best efforts: in the world of
customer centricity, there are good customers…and then there is
pretty much everybody else.”
Anti-Pattern Hypotheses:
● “We’ll launch this feature as long as long as it doesn’t tank the site.” -- (Smoke Tests)
○ Better: Not always bad, but keep minimal in your testing portfolio.
○ “We’ll launch this feature as long as users respond at least neutrally according
to X KPI.”
● “We will know we are successful when conversion increases x% with this test.”
○ Better: “Same-session conversion should increase x% with this test for this
segment of users for this variation. For other segments, another variation will
drive y% increase in same-session conversion. This is because returning
travelers will have more information in the email to make a decision more
quickly.”

39. Principle 5: Avoid Product-Centric mentality
- focus on User-Centric approaches
Peter Fader - Customer Centricity: Focus on the Right Customers for
Strategic Advantage:
“Not all customers deserve your best efforts: in the world of
customer centricity, there are good customers…and then there is
pretty much everybody else.”
Anti-Pattern Hypotheses:
● “We’ll launch this feature as long as long as it doesn’t tank the site.” -- (Smoke Tests)
○ Better: Not always bad, but keep minimal in your testing portfolio.
○ “We’ll launch this feature as long as users respond at least neutrally according to X KPI.”
● “We will know we are successful when conversion increases x% with this test.”
○ Better: “Same-session conversion should increase x% with this test for this segment of
users for this variation. For other segments, another variation will drive y% increase in same-
session conversion. This is because returning travelers will have more information in the
email to make a decision more quickly.”
● “We should improve the user experience for everyone.”

40. Principle 5: Avoid Product-Centric mentality
- focus on User-Centric approaches
Peter Fader - Customer Centricity: Focus on the Right Customers for
Strategic Advantage:
“Not all customers deserve your best efforts: in the world of
customer centricity, there are good customers…and then there is
pretty much everybody else.”
Anti-Pattern Hypotheses:
● “We’ll launch this feature as long as long as it doesn’t tank the site.” -- (Smoke Tests)
○ Better: Not always bad, but keep minimal in your testing portfolio.
○ “We’ll launch this feature as long as users respond at least neutrally according to X KPI.”
● “We will know we are successful when conversion increases x% with this test.”
○ Better: “Same-session conversion should increase x% with this test for this segment of
users for this variation. For other segments, another variation will drive y% increase in same-
session conversion. This is because returning travelers will have more information in the
email to make a decision more quickly.”
● “We should improve the user experience for everyone.”
○ Better: “We should optimize towards our targeted demographic - our most valuable users.”

41. Conclusion: Print Out This Slide!
1. Averages are your enemy - focus on time series
1. Beware of perverse incentives - focus on the bigger picture
1. Everything is a bet - focus on adaptability
1. Beware of post-hoc storytelling - focus on a priori hypothesizing
1. Avoid Product-centric mentality - focus on User-centric
approaches