Kyle Rush, Head of Optimization at Optimizely Benn Stancil, Co-Founder and Chief Analyst at Mode In this session Kyle Rush, Head of Optimization at Optimizely, and Benn Stancil, Co-Founder and Chief Analyst at Mode, discuss how to develop a strategy to get the most out of your test results. Kyle has conducted hundreds of experiments on web properties like Obama for America, Mozilla, and Optimizely, while Benn's expertise is drawn from experiences at Yammer, Microsoft, and data analysis in other fields. This combined knowledge serve as the basis for this session's discussion, which includes micro and macro strategy, what can be learned from experiment results, and suggestions for what you should test.

1. Optimizing
Your
Optimization
Strategy
Benn
Stancil
Co-­‐Founder
and
Chief
Analyst
Mode
@bennstancil
Kyle
Rush
Head
of
Optimization
Optimizely
@kylerush

2. Focus
on
high
value
flows

3. What
to
test:
focus
!
HIGH
VALUE
FLOWS
vs.
ONE
OFFS

4. What
to
test:
focus

7. Define
success
before
you
start
Track
many
goals

8. This
is
what
16
goals
looks
like

9. Define
success
before
you
start
But
choose
one
primary
goal.
The
right
primary
goal.

10. What
to
test:
geJng
ideas
Comb
through
your
data.
17%
error
rate

11. What
to
test:
geJng
ideas
Submit Success
a
b
a
b
3.5%
absolute
difference.
26.3%
relaNve
difference.

12. What
to
test:
geJng
ideas
Go
through
the
flow
yourself.
100
Nmes.

13. What
to
test:
geJng
ideas
CompeNtor
research

15. What
to
test:
geJng
ideas
Doesn’t
have
to
be
a
compeNtor

17. What
to
test:
geJng
ideas
User
tesNng
•
Talk
to
friends
*She
misses
you.
•
Call
your
mom*
•
Talk
to
peers

18. Your
strategy
should
revolve
around
a
variable
called
MDE.
(minimum
detectable
effect)
Sample
size

19. Everyone
wants
a
low
MDE.
But
most
of
us
don’t
have
the
luxury.
Sample
size

20. Myth:
Small
startups
can’t
a/b
test
Your
small
website
gets
2,000
visitors
a
month
Sample
size

21. Sample
size
calculator
h_p://www.evanmiller.org/ab-­‐tesNng/sample-­‐size.html

22. Significance?
Power?
Sample
size
Significance:
5%,
Power:
80%

23. Determine
sample
size
before
starNng
the
experiment

24. Sample
size
How
big
is
the
decision
the
test
will
help
decide?

25. A
couple
tricks
for
a
lower
MDE
TargeOng

26. Not
signed
in
4.2%
conversion
rate
Signed
in
0.06%
conversion
rate
average
conversion
rate:
2.3%*
*the
“not
signed
in”
to
“signed
in”
visits
raNo
is
not
50/50,
but
i’m
saying
it
is
for
the
purpose
of
this
example.

27. Almost
twice
as
many
visitors
would
be
needed.
Not
signed
in Signed
in

28. So
you
want
a
lower
MDE?
TargeOng
Here’s
your
chance.

29. 36%
lower
MDE
:)
Not
signed
in Signed
in

30. Another
Np
for
a
lower
MDE
TargeOng

31. conversion
rate:
4.2% conversion
rate:
6.7%

32. 22%
lower
MDE
:)
Bad
targeNng Good
targeNng

33. OpNmizing
your
opNmizaNon
strategy
Benn
Stancil
Co-­‐Founder
and
Chief
Analyst
at
Mode
@bennstancil

34. Previous Now

35. OUTLINE
Test to Learn
Testing is not just for making
individual decisions
!

36. OUTLINE
Test to Learn
Testing is not just for making
individual decisions
!
Know What You Don’t Know
Be aware of what you can’t see
!

37. OUTLINE
Test to Learn
Testing is not just for making
individual decisions
!
Know What You Don’t Know
Be aware of what you can’t see
!
Think Beyond One Test
Aim, fire, adjust

38. Before Testing:
!
TEST TO LEARN

39. TEST TO LEARN
“Give a man a fish…”

40. TEST TO LEARN
User research has drawbacks

41. TEST TO LEARN
Why we need controlled experiments

42. TEST TO LEARN
Why don’t new users post?

43. TEST TO LEARN
Hard questions don’t require hard tests

44. TEST TO LEARN
Do people respond
to features that are
“coming soon”?

45. While Testing:
!
KNOW WHAT YOU DON’T KNOW

46. KNOW WHAT YOU DON’T KNOW

47. KNOW WHAT YOU DON’T KNOW
Analyzing test results isn’t just interpreting what
you can from results.
!
It’s also knowing what the test doesn’t tell you.

48. KNOW WHAT YOU DON’T KNOW
Yadier Molina Ivan Rodriguez

49. KNOW WHAT YOU DON’T KNOW
Why do new customers test differently?

50. KNOW WHAT YOU DON’T KNOW

51. KNOW WHAT YOU DON’T KNOW
•Be skeptical
!
•Don’t leave out details
!
•Be comfortable with caveats

52. After Testing:
!
THINK BEYOND ONE TEST

53. THINK BEYOND ONE TEST
Aim, fire, adjust

54. THINK BEYOND ONE TEST
Ask why

55. THINK BEYOND ONE TEST
Yammer’s new user badge

56. THINK BEYOND ONE TEST
Keep thorough records

57. THINK BEYOND ONE TEST
Getting feeds just right

58. YOUR OPTIMIZATION STRATEGY
Test to Learn
Testing is not just for making
individual decisions
!
Know What You Don’t Know
Be aware of what you can’t see
!
Think Beyond One Test
Aim, fire, adjust

59. Optimizing
Your
Optimization
Strategy
Benn
Stancil
Co-­‐Founder
and
Chief
Analyst
Mode
@bennstancil
Kyle
Rush
Head
of
Optimization
Optimizely
@kylerush