Optimizely launched our new Stats Engine in January 2015 to power results for all A/B and Multivariate tests run on the platform. Stats Engine lets experimenters see results that are always statistically valid so they can make decisions in real time. These slides provide an overview of why Optimizely made our new Stats Engine and the problems with traditional statistics that Stats Engine solves. It also provides an introduction to the methods behind Stats Engine, sequential testing and false discovery rate control. Finally, the slides give practical recommendations for how to run an A/B or multivariate test with Stats Engine, and how to decide when you might want to stop your test. The slides were presented by Darwish Gani, product manager, and Leo Pekelis, statistician, at Optimizely. See a full recording of the webinar in Optiverse: https://community.optimizely.com/t5/Presentations/Webinar-recording-Stats-Engine-Q-amp-A-webinar-recording-and/ba-p/9776

1. Optimizely Stats Engine
Leo Pekelis
Darwish Gani
Robin Pam

2. Housekeeping notes
• Chat box is available for questions
• There will be time for Q&A at the end
• We will be recording the webinar for future viewing
• All attendees will receive a copy of slides after the
webinar

3. Your speakers
Darwish Gani
Product manager
Robin Pam
Product marketing
Leo Pekelis
Statistician

4. Objectives
Understand why Optimizely built Stats Engine
Introduce the methods Stats Engine uses to
calculate results
Get practical recommendations for how to test with
Stats Engine

5. Why make a new Stats Engine?

6. Meet Joe: A farmer who uses a traditional t-test

7. Joe wants to try a new fertilizer this year

8. With his original fertilizer, 10% of plants
survive the winter
He thinks that this new fertilizer might help
more survive.
Joe has a hypothesis

9. Joe calculates a sample size for his experiment in advance,
given how much better he thinks the new fertilizer might be

10. He waits through the winter…

11. 10% of plants survive the winter
15% of plants survive
96% statistical significance!
…and he is rewarded for his patience

12. Does that work today?

13. Meet Kyle: Head of Optimization at Optimizely

14. Kyle doesn’t know what improvement to expect

15. Kyle also gets data from Optimizely all the time

16. Kyle wants to test many goals and variations at
once, instead of just one hypothesis
vs.

17. Actually that’s a lot of work.
It’s cumbersome and error-prone.

18. What’s your chance of making incorrect decision?

19. 30%

20. Objectives
Understand why Optimizely built Stats Engine
Introduce the methods Stats Engine uses to
calculate results
Get practical recommendations for how to test with
Stats Engine

21. Introducing Stats Engine

22. How we did it
• Partnered with Stanford
statisticians
• Talked with customers
• Examined historical
experiments
• Found the best methods
for real-time data

23. What does Stats Engine do?
Provides a principled and mathematical way to calculate your
chance of making an incorrect decision.

24. Sequential Testing False Discovery Rate
• First used in 1940s for military
weapons testing
• Sample size is not fixed in
advance
• Data is evaluated as it’s collected
• First used in 1990s in genetics
• Correct error rates for
multiple goals and variations
• Expected number of false
discoveries

25. Sequential Testing
False Discovery Rate control
+
=
Statistical Significance for Digital Experimentation

26. Statistical Significance for Digital Experimentation
Continuously Evaluate Test Results
Run many goals and variations
Don’t worry about estimating a
MDE upfront

27. Sequential Testing
Finding the right stopping rule

28. Variation #1
Variation #2
Declare a winner?
500
Visitors
50%
65%
Is this lift big enough
for the visitors I saw?
?

29. Desired Stopping Rule:
I will be “wrong” only 5% of the time.

30. Variation #1
Variation #2
1000
Visitors
55%
59%
Traditional Error Rates 5%
Find a stopping rule, so I declare
a winner incorrectly <5% of the
time

31. Variation #1
Variation #2
Traditional Error Rates
Visitors
500
50%
65%
5%
1000
55%
59%
5%
5000
52%
57%
5%
10000
54%
59%
5%

32. Variation #1
Variation #2
Traditional Error Rates
500 1000 5000 10000
Visitors
50% 55% 52% 54%
65% 59% 57% 59%
5% 5% 5% 5%
Look only once: 5% Error rate

33. Variation #1
Variation #2
Traditional Error Rates
500 1000 5000 10000
Visitors
50% 55% 52% 54%
65% 59% 57% 59%
5% 5% 5% 5%
Look only once: 5% Error rate
Look more than once: >5% Error rate

34. Variation #1
Variation #2
Traditional Error Rates
500 1000 5000 10000
Visitors
50% 55% 52% 54%
65% 59% 57% 59%
5% 5% 5% 5%
<5% Error rate for the entire test!
Sequential Testing
Error Rate
1% .5% 1.5% 1.5% < 5%

35. “P-Hacking”
“Continuous Monitoring”
“Repeated Significance Testing”
These are not new problems!

36. Steven Goodman, Stanford Physician & Statistician, nature.com
“The P value was never meant to be used the
way it's used today.”

37. Source: Evan Miller, How not to Run an A/B Test

39. Sample Size + Power Calculations
Focus on creating and running tests

40. Sequential Testing
• Continuously Evaluate Test Results
• Don’t worry about estimating a
MDE upfront
Framework of hypothesis testing that was created to allow the
experimenter to evaluate test results as they come in

41. False discovery rate control
Error rates for a world with many goals and
variations

42. Variations Goal 1 Goal 2 Goal 3 Goal 4 Goal 5
Control
Variation 1
Variation 2

43. Variations Goal 1 Goal 2 Goal 3 Goal 4 Goal 5
Control
Variation 1
Variation 2
Significance Level 90 (False Positive Rate 10%)

44. Variations Goal 1 Goal 2 Goal 3 Goal 4 Goal 5
Control
Variation 1
Variation 2
Significance Level 90 (False Positive Rate 10%)
1 False Positive!

45. Variations Goal 1 Goal 2 Goal 3 Goal 4 Goal 5
Control
Variation 1
Variation 2
1 False Positive!
Significance Level 90 (False Positive Rate 10%)
1 other variation x goal has a large improvement.

46. Variations Goal 1 Goal 2 Goal 3 Goal 4 Goal 5
Control
Variation 1
Variation 2
1 False Positive!
Significance Level 90 (False Positive Rate 10%)
1 other variation x goal has a large improvement.
1 True Positive!

47. My Report
• Variation 2 is improving on Goal 1
• Variation 1 is improving on Goal 4
“10% of what I
report could be wrong.”
X
50%
• Variation 2 is improving on Goal 1
• Variation 1 is improving on Goal 4
Furthermore, I found the following results.
This leads me to conclude that …

48. Statistical Significance:
The chance that any variation on any goal
that is reported as a winner or loser
is correct.

49. • “New York Times has a feature in its Tuesday science section, Take a Number … Today’s
column is in error … This is the old, old error of confusing p(A|B) with p(B|A).”
• Andrew Gelman, Misunderstanding the p-value
• “If I were to randomly select a drug out of the lot of 100, run it through my tests, and
discover a p<0.05 statistically significant benefit, there is only a 62% chance that the
drug is actually effective.”
• Alex Reinhart, The p value and the base rate fallacy
• “In this article I’ll show that badly performed A/B tests can produce winning results
which are more likely to be false than true. At best, this leads to the needless
modification of websites; at worst, to modification which damages profits.”
• Martin Goodson, Most Winning A/B Test Results are Illusory
• “An unguarded use of single-inference procedures results in a greatly increased false
positive (significance) rate”
• Benjamini, Yoav, and Yosef Hochberg. "Controlling the false discovery rate: a practical and powerful approach to
multiple testing." Journal of the Royal Statistical Society. Series B (Methodological) (1995): 289-300. APA

50. • “New York Times has a feature in its Tuesday science section, Take a Number … Today’s
column is in error … This is the old, old error of confusing p(A|B) with p(B|A).”
• Andrew Gelman, Misunderstanding the p-value
• “If I were to randomly select a drug out of the lot of 100, run it through my
tests, and discover a p<0.05 statistically significant benefit, there is only a
62% chance that the drug is actually effective.”
• Alex Reinhart, The p value and the base rate fallacy
• “In this article I’ll show that badly performed A/B tests can produce winning results
which are more likely to be false than true. At best, this leads to the needless
modification of websites; at worst, to modification which damages profits.”
• Martin Goodson, Most Winning A/B Test Results are Illusory
• “An unguarded use of single-inference procedures results in a greatly increased false
positive (significance) rate”
• Benjamini, Yoav, and Yosef Hochberg. "Controlling the false discovery rate: a practical and powerful approach to
multiple testing." Journal of the Royal Statistical Society. Series B (Methodological) (1995): 289-300. APA

51. False Discovery Rate control
Framework for controlling errors that arise from running multiple
experiments at once.
Run many goals & variations

52. What Stats Engine means for you
• You see
fewer, but more accurate conclusive results.
• You can
implement winners as soon as significance is reached.
• You get
• easy experiment workflow.
• reduced unforeseen, and hidden errors.

53. Objectives
Understand why Optimizely built Stats Engine
Introduce the methods Stats Engine uses to
calculate results
Get practical recommendations for how to test with
Stats Engine

54. And now, for some practical guidance

55. First, some vocabulary
• Baseline conversion rate
The control group’s expected conversion rate.
• Minimum detectable effect
The smallest conversion rate difference it is
possible to detect in an A/B Test.
• Statistical significance
The likelihood that the observed difference in
conversion rates is not due to chance.
• Minimum sample size
The smallest number of visitors required to
reliably detect a given conversion rate
difference

56. Sample size calculators and statistical power

57. How many visitors do you need to see significant
results?
Visitors needed to reach significance
with Stats Engine
Improvement
5% 10% 25%
Baseline
conversion rate
1% 458,900 101,600 13,000
5% 69,500 15,000 1,800
10% 29,200 6,200 700
25% 8,100 1,700 200
Lower conversion rate, lower effects = more visitors

58. One example of calculating your opportunity cost
12% minimum
detectable effect

59. Original
Variation

60. Should you stop or continue a test?

61. Should you stop or continue a test?
Is my test
significant?
Congrats
Can I afford
to wait?
Continue
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
No
Yes

62. Should you stop or continue a test?
Is my test
significant?
Congrats
Can I afford
to wait?
Continue
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
No
Yes

64. Difference intervals

65. Difference intervals
15.4
11.6Middle Ground
Best Case
Worst case
7.3

66. Seasonality
• We DO take into account seasonality while a test is
running.
• We DO NOT take into account future seasonality after an
experiment is stopped.

67. Should you stop or continue a test?
Is my test
significant?
Congrats
Can I afford
to wait?
Continue
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
No
Yes
• Use Difference
Intervals to understand
the types of lifts you
could see.

68. Should you stop or continue a test?
Is my test
significant?
Congrats
Can I afford
to wait?
Continue
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
No
Yes

69. A reasonable number of visitors left, relative to
your traffic?

70. Visitors Remaining Explained
Improvement
Increases in
Magnitude
Improvement
Stays the Same
Improvement
Decreases in
Magnitude

71. A good idea to wait
5761 + 3200
Note!

72. Should you stop or continue a test?
Is my test
significant?
Congrats
Can I afford
to wait?
Continue
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
No
Yes
• Use Visitors Remaining
to evaluate if waiting
makes sense.

73. Should you stop or continue a test?
Is my test
significant?
Congrats
Can I afford
to wait?
Continue
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
No
Yes

75. If you’re an organization that can
• Iterate quickly on new variations
• Run lots of experiments
• Have little downside risk of implementing non-winning
variations
then you can likely tolerate a higher error rate.

76. Difference intervals can guide your decision

77. Should you stop or continue a test?
Is my test
significant?
Congrats
Can I afford
to wait?
Continue
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
No
Yes • Use Difference
Intervals to measure
risk you take on

78. > 100,000 visitors? What next?

79. Time to move on to the next idea

80. Should you stop or continue a test?
Is my test
significant?
Congrats
Can I afford
to wait?
Continue
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
No
Yes
• Use Visitors Remaining
to evaluate if waiting
makes sense.

81. Recap
Is my test
significant?
Congrats
Can I afford
to wait?
Continue
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
No
Yes
• Use Difference
Intervals to understand
the types of lifts you
could see.

82. Recap
Is my test
significant?
Congrats
Can I afford
to wait?
Continue
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
No
Yes
• Use Difference
Intervals to understand
the types of lifts you
could see.
Can I afford
to wait?
Continue
Stop
Concede
inconclusive
Yes
No
No
• Use Visitors Remaining
to evaluate if waiting
makes sense.

83. Recap
Is my test
significant?
Congrats
Can I afford
to wait?
Continue
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
No
Yes
• Use Difference
Intervals to understand
the types of lifts you
could see.
Can I afford
to wait?
Continue
Stop
Concede
inconclusive
Yes
No
No
• Use Visitors Remaining
to evaluate if waiting
makes sense.
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
Can I afford
to wait?
• Use Difference
Intervals to measure
risk you take on

84. Recap
Is my test
significant?
Congrats
Can I afford
to wait?
Continue
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
No
Yes
• Use Difference
Intervals to understand
the types of lifts you
could see.
Can I afford
to wait?
Continue
Stop
Concede
inconclusive
Yes
No
No
• Use Visitors Remaining
to evaluate if waiting
makes sense.
Stop
Accept lower
significance
Concede
inconclusive
Yes
No
Can I afford
to wait?
• Use Difference
Intervals to measure
risk you take on
• Use Visitors Remaining
to evaluate if waiting
makes sense.

85. Tuning your testing strategy for your traffic and
business
1%6% 4% 3%
inconclusive inconclusive inconclusiveinconclusive 18%
Looking for small effects?
• Tests take longer to reach
significance
• Might find more winners, if you
are willing to wait long enough
inconclusive
Testing for larger effects?
• Run more tests, faster
• Know when it’s time to move on
to the next idea

86. Frequently asked questions

87. Do I need to re-run my historical tests?

88. Is this a one-tailed or two-tailed test? Why
did you switch?

89. Why does Stats Engine report 0%
Statistical Significance when other tools
report higher values?

90. Why does Statistical Significance increase
step-wise?

91. If I take the results I see in Optimizely and
plug them into any other statistics
calculator, the statistical significance is
different. Why?

92. How does Stats Engine handle revenue
calculations?

93. Questions?