Weitere ähnliche Inhalte Ähnlich wie Jesse Hull - LOGIN Conference 2012 (20) Jesse Hull - LOGIN Conference 20121. Building a Culture of A/B Testing
Jesse Hull
Executive Director of Technology, Social Games
GSN Digital
4. What is A/B Testing? - Randomly Assign Users
We are the
same
Group A Group B
Group A Group B
4
5. What is A/B Testing? – Show Different Content
A
B Group A Group B
5
7. “”A/B testing is like sandpaper. You
can use it to smooth out details, but
you can't actually create anything
with it.”
7
9. You Can Do A Lot More Then Just
Smooth Out The Details With…
Holistic A/B Testing
9
12. And They Come Up With Tests Like
These
A B
(2.4% more revenue)
12
23. Video Bingo Won
• Time playing: +356% (over closest competition)
• Next day return rate: +42%
• Revenue: +238%
23
25. • Time spent playing: +12.9%
• Next day return rate: +6.8%
• Revenue: +3.6%
25
34. How Do You Build An Organization That
Can Holistically A/B Test?
In 2 Easy Steps…
34
38. Set Team Goals In Terms Of Tests
Perform 10 A/B tests focused on
improving daily active users.
The combination of these tests should
improve DAU's by 20%.
38
43. Everyone Is Testing: Artists
89.3% 4.3% 23.5% 42.0% 8.9%
click rate install rate click rate click rate install rate
12,039 130.2% 0.4% 34,084 41.2%
publishes click rate install rate publishes click rate
43
45. Everyone Is Testing: Marketing
• Test sending emails to only 50% of your user
base
• Cross-promotion
45
46. Deciding What To Test
• Hire great people
• Improve their intuition by testing
46
47. When Not to A/B Test
Aw, come on, baby. I wasn’t really cheating on you I was A/B testing.
47
48. When Not to A/B Test
• Technology limitations
• No users
• Long-term effects
48
50. the impact of your testing
is limited only by the
breadth and creativity of
the ideas you test.
50
52. "Do not believe in anything simply because
you have heard it. Do not believe in
anything simply because it is spoken and
rumored by many.... But after observation
and analysis, when you find that anything
agrees with reason and is conducive to the
good and benefit of one and all, then accept
it and live up to it."
-- Buddha
52