Jesse Hull - Building A Culture of A/B Testing - LOGIN Conference 2012 - 9/26/2012

1. Building a Culture of A/B Testing
Jesse Hull
Executive Director of Technology, Social Games
GSN Digital

2. 2

3. 3

4. What is A/B Testing? - Randomly Assign Users
We are the
same
Group A Group B
Group A Group B
4

5. What is A/B Testing? – Show Different Content
A
B Group A Group B
5

6. What is A/B Testing? – Measure Result
A B
6

7. “”A/B testing is like sandpaper. You
can use it to smooth out details, but
you can't actually create anything
with it.”
7

8. This Is Wrong!
8

9. You Can Do A Lot More Then Just
Smooth Out The Details With…
Holistic A/B Testing
9

10. What is Holistic A/B Testing?
Principle #1
Everyone is Testing
10

11. Most Organizations Use These
People To Test
11

12. And They Come Up With Tests Like
These
A B
(2.4% more revenue)
12

13. But You Want These People Testing
13

14. What is Holistic A/B Testing?
Principle #2
Test Big and Small
Ideas
14

15. Why Small Ideas?
10 Ideas That Each Improve Revenue 4%
+50%
15

16. Why Big Ideas?
16

17. Example:
Casino Games
17

18. Game Designer
Tested Wildly Different Casino Game Mechanics
18

19. Scratch Madness
19

20. Roulette
20

21. Video Bingo
21

22. Blackjack
22

23. Video Bingo Won
• Time playing: +356% (over closest competition)
• Next day return rate: +42%
• Revenue: +238%
23

24. That Was Just The Beginning
Engineers
Improved Load Time
24

25. • Time spent playing: +12.9%
• Next day return rate: +6.8%
• Revenue: +3.6%
25

26. Artists
Tested Completely New UI
26

27. 27

28. 28

29. • Revenue: +12.3%
• Bounce Rate: -25.8%
29

30. Enterprising Developer
Tested Various Power-Ups
30

31. Three Extra Balls At The End
Revenue: +5.9%
31

32. U Pick’em
Revenue: +4.7%
32

33. • Revenue: +23.6%!
• Next day return: unchanged
33

34. How Do You Build An Organization That
Can Holistically A/B Test?
In 2 Easy Steps…
34

35. Organization: Step 1
Build A Framework
That Everyone Can
Use
35

36. A/B Testing Framework
Variations
Keep
Common
Testing
Infrastructure
36

37. Organization: Step 2
Build A/B Testing Into
Your Culture
37

38. Set Team Goals In Terms Of Tests
Perform 10 A/B tests focused on
improving daily active users.
The combination of these tests should
improve DAU's by 20%.
38

39. Everyone Is Testing: Game
Designers
39

40. Everyone Is Testing: Game
Designers
40

41. Everyone Is Testing: Game
Designers
41

42. Everyone Is Testing: Artists
42

43. Everyone Is Testing: Artists
89.3% 4.3% 23.5% 42.0% 8.9%
click rate install rate click rate click rate install rate
12,039 130.2% 0.4% 34,084 41.2%
publishes click rate install rate publishes click rate
43

44. Everyone Is Testing: Engineers
• Performance improvements
• Architectural changes
44

45. Everyone Is Testing: Marketing
• Test sending emails to only 50% of your user
base
• Cross-promotion
45

46. Deciding What To Test
• Hire great people
• Improve their intuition by testing
46

47. When Not to A/B Test
Aw, come on, baby. I wasn’t really cheating on you I was A/B testing.
47

48. When Not to A/B Test
• Technology limitations
• No users
• Long-term effects
48

49. Everyone in the
organization should be
testing all the time
because...
49

50. the impact of your testing
is limited only by the
breadth and creativity of
the ideas you test.
50

51. Our Company
• Contact Me: jhull@gsn.com
We Are Hiring!
www.gsngames.com
51

52. "Do not believe in anything simply because
you have heard it. Do not believe in
anything simply because it is spoken and
rumored by many.... But after observation
and analysis, when you find that anything
agrees with reason and is conducive to the
good and benefit of one and all, then accept
it and live up to it."
-- Buddha
52

53. Questions?
53