This presentation was given at the Optimizely Experience New York City by data scientist, Kaiser Fung. The title of his presentation is 4 questions about testing you wanted to ask but didn't.

1. KAISER FUNG
Aug. 26, 2014
New York City
kaiserdatamatter@gmail
Q4
~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung

2. Testing has played a big part in my career
(c) Kaiser Fung

3. Q4
~ Testing
U wanted 2
But Didn’t
1. Why worry about
statistical significance?
2. Can I use a simple
sample size formula?
3. Is 90% confidence
too risk-averse?
4. What is the value of
testing?
(c) Kaiser Fung

4. Varying Copy on a Conversion Page
(c) Kaiser Fung

5. Not Statistically Significant
(c) Kaiser Fung

6. Q*Q3
~ Testing
U wanted 2
But Didn’t
1
Why should
we care about
statistical
significance?
(c) Kaiser Fung

7. (c) Kaiser Fung

8. (c) Kaiser Fung

9. Statistical Noise
(c) Kaiser Fung

10. (c) Kaiser Fung

11. (c) Kaiser Fung

12. (c) Kaiser Fung

13. (c) Kaiser Fung

14. Statistical Noise
(c) Kaiser Fung

15. Icon by Laura Begg
40,000
(c) Kaiser Fung

16. 20,000
(c) Kaiser Fung

17. 30,000
(c) Kaiser Fung

18. 50:50
(c) Kaiser Fung

19. 15k 15k (c) Kaiser Fung

20. 125
(c) Kaiser Fung

21. 2
Can we use a
simple sample
sizing formula?
Q*Q*Q2
~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung

22. 10%90%
(c) Kaiser Fung

23. More Samples
More Precision
(c) Kaiser Fung

24. (c) Kaiser Fung

25. 3
Is 90% confidence
too risk averse?
Q2*Q*Q
~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung

26. OMG
(c) Kaiser Fung

27. 90% Confidence means:
If the analyst tells me the test is
significant, there is a 90% chance
this effect is really there
(c) Kaiser Fung

28. 90% Confidence means:
If the analyst tells me the test is
significant, there is a 90% chance
this effect is really there
Good Question – Wrong Definition
(c) Kaiser Fung

29. 90% Confidence means:
If the effect is really there, there is a
90% chance the analyst will tell me
the test is significant
(c) Kaiser Fung

30. 90% Confidence means:
If the effect is really there, there is a
90% chance the analyst will tell me
the test is significant
Correct Definition – Wrong Question
(c) Kaiser Fung

31. Positive Predictive Value:
If the analyst tells me the test is
significant, what is the probability
this effect is really there?
(c) Kaiser Fung

32. 90% Confidence =
63% PPV
(c) Kaiser Fung

33. PPV = 45-60%
(c) Kaiser Fung

34. Chapter 4:
Timid Testers /
Magic Lassos
(c) Kaiser Fung

35. 4
What’s the value
of testing?
Q3*Q
~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung

36. BEFORE AFTER
xkcd (c) Kaiser Fung

37. B.T.
30%
70% 63%
37%
A.T.
(c) Kaiser Fung

38. 1. Why statistical
significance
2. How sample
sizing
3. What
confidence level
4. Why test
Statistical Noise
Safe Landing
Measure of Risk
PPV Lift
A/A Test
Sq-root Law
PPV
Bayes
Q4
~ Testing
U wanted 2
But Didn’t
(c) Kaiser Fung

39. KAISER FUNG
Strategic data advisory services,
including test planning & analysis
Speaking on business analytics & dataviz
Training in statistical thinking
(c) Kaiser Fung

40. Thank you
Kaiser Fung
kaiserdatamatter@gmail.com
Twitter: @junkcharts
LinkedIn
(c) Kaiser Fung