Kyle Rush, Head of Optimization at Optimizely
Benn Stancil, Co-Founder and Chief Analyst at Mode
In this session Kyle Rush, Head of Optimization at Optimizely, and Benn Stancil, Co-Founder and Chief Analyst at Mode, discuss how to develop a strategy to get the most out of your test results.
Kyle has conducted hundreds of experiments on web properties like Obama for America, Mozilla, and Optimizely, while Benn's expertise is drawn from experiences at Yammer, Microsoft, and data analysis in other fields. This combined knowledge serve as the basis for this session's discussion, which includes micro and macro strategy, what can be learned from experiment results, and suggestions for what you should test.
Targeted Rollouts: How to Release Features to Multiple Audiences
Optimizing Your Optimization Strategy
1. Optimizing
Your
Optimization
Strategy
Benn
Stancil
Co-‐Founder
and
Chief
Analyst
Mode
@bennstancil
Kyle
Rush
Head
of
Optimization
Optimizely
@kylerush
26. Not
signed
in
4.2%
conversion
rate
Signed
in
0.06%
conversion
rate
average
conversion
rate:
2.3%*
*the
“not
signed
in”
to
“signed
in”
visits
raNo
is
not
50/50,
but
i’m
saying
it
is
for
the
purpose
of
this
example.
27. Almost
twice
as
many
visitors
would
be
needed.
Not
signed
in Signed
in
28. So
you
want
a
lower
MDE?
TargeOng
Here’s
your
chance.
36. OUTLINE
Test to Learn
Testing is not just for making
individual decisions
!
Know What You Don’t Know
Be aware of what you can’t see
!
37. OUTLINE
Test to Learn
Testing is not just for making
individual decisions
!
Know What You Don’t Know
Be aware of what you can’t see
!
Think Beyond One Test
Aim, fire, adjust
47. KNOW WHAT YOU DON’T KNOW
Analyzing test results isn’t just interpreting what
you can from results.
!
It’s also knowing what the test doesn’t tell you.
48. KNOW WHAT YOU DON’T KNOW
Yadier Molina Ivan Rodriguez
49. KNOW WHAT YOU DON’T KNOW
Why do new customers test differently?
58. YOUR OPTIMIZATION STRATEGY
Test to Learn
Testing is not just for making
individual decisions
!
Know What You Don’t Know
Be aware of what you can’t see
!
Think Beyond One Test
Aim, fire, adjust
59. Optimizing
Your
Optimization
Strategy
Benn
Stancil
Co-‐Founder
and
Chief
Analyst
Mode
@bennstancil
Kyle
Rush
Head
of
Optimization
Optimizely
@kylerush