Weitere ähnliche Inhalte Ähnlich wie A/B Testing - What your mother didn't tell you (20) Kürzlich hochgeladen (20) A/B Testing - What your mother didn't tell you2. A/B
TesAng
What
your
mother
didn’t
tell
you
hDp://www.allaroundtheworld.fr/
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
3. What
is
A/B
TesAng?
• TesAng
customers,
not
code
• Show
different
versions
• Pick
the
best
one
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
4. Who
Relies
On
A/B
TesAng?
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
5. Transform
Your
Company
• Test
your
customers,
not
your
code
• Stop
guessing
what
works
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
9. A
Basic
Test
•
•
•
•
Your
current
behavior
is
the
base
A
variant
is
a
new
version
( just
one
change)
You
can
have
one
or
more
variants
But
we
assume
only
one
for
this
talk
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
12. Which
Generates
More
Clicks?
10,017
Visits
125
Conversions
January
14,
2014
9,983
Visits
132
Conversions
©
2014
hDp://www.allaroundtheworld.fr/
13. Which
Generates
More
Clicks?
9,950
Visits
125
Conversions
January
14,
2014
10,550
Visits
155
Conversions
©
2014
hDp://www.allaroundtheworld.fr/
14. The
Numbers
(95%
Confidence)
Version
Visits
Conversion
Rate
Not
Random
Red
10,017
125
1.25%
—
Blue
9,983
132
1.32%
51%
hDp://getdelve.com/2011/12/math-‐behind-‐ab-‐tesAng-‐for-‐curious-‐execuAves-‐part-‐1/
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
15. The
Numbers
(95%
Confidence)
Version
Visits
Conversion
Rate
Not
Random
Red
9,950
125
1.26%
—
Blue
10,550
155
1.47%
94%
hDp://getdelve.com/2011/12/math-‐behind-‐ab-‐tesAng-‐for-‐curious-‐execuAves-‐part-‐1/
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
16. Which
Sells
More
Product?
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
21. Roll
the
Dice
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
23. Choose
the
Best
Pa:ence
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
25. Choose
the
Best
•
•
•
•
Ignore
the
first
day
Full-‐business
cycle
What
if
you’re
wrong?
False
posiAves
hDp://www.einarsen.no/is-‐your-‐ab-‐tesAng-‐effort-‐just-‐chasing-‐staAsAcal-‐ghosts/
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
26. LimitaAons
of
A/B
TesAng
• Doesn’t
tell
you
why
• Side
effects
• Doesn’t
tell
you
loyalty
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
27. Benefits
•
•
•
•
•
•
•
•
Hyundai:
62%
increase
in
test
drive
requests
Artsy
Editor:
47%
increase
in
click-‐throughs
Visual
Website
OpAmizer:
50%
signup
increase
AssessmentDay:
62%
increase
in
sales
SmileyCookie:
41%
increase
in
sales
BaseKit:
25%
increase
in
conversions
Mobal:
27%
increase
in
sales
You
Need
A
Budget:
85%
increase
in
downloads
hDp://visualwebsiteopAmizer.com/case-‐studies.php
28. What
Are
They
TesAng?
•
•
•
•
•
•
•
•
Add-‐to-‐cart
buDon
placement
Call
to
acAon
buDons
Using
popup
forms
Removing
content
Link
color
Adding
tesAmonials
NavigaAon
menus
Removing
signup
forms
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
29. Gemng
Started
•
•
•
•
Vendor
or
in-‐house?
Go
for
quick
wins
Run
an
AA
test
Be
pa:ent!
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
31. Problems
•
•
•
•
•
•
•
Forgemng
caching
Non-‐random
selecAon
Forgemng
selecAon
TesAng
only
the
view
Performance
Dead
code
Profit
versus
conversion
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
32. Tips
•
•
•
•
•
Most
experiments
fail
Don’t
roll
the
dice
prematurely
A/B
test
bugs
Test
“stupid”
ideas
Don’t
require
rollouts!
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
33. SegmentaAon
•
•
•
•
•
•
New
versus
returning
customers
Different
browsers
Accept-‐Language
Time
of
day
IP
country
Gender?
Age?
Something
else?
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
34. See
Also
• The
MulA-‐Armed
Bandit
hDp://stevehanov.ca/blog/index.php?id=132
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/
35. Conclusion
•
•
•
•
•
Lots
of
money
lying
on
the
table
Egos
are
bad
Data
is
good
Customer
behavior
trumps
expert
opinion
Customer
behavior
trumps
code
behavior
January
14,
2014
©
2014
hDp://www.allaroundtheworld.fr/