HOW TO BUILD A SUCCESSFUL EXPERIMENTATION PROGRAM

1. HOW TO BUILD A SUCCESSFUL
EXPERIMENTATION PROGRAM
Gokce Tombul

2. A little bit about me
Gokce Tombul
Sr. Director of Product, Growth & Discovery

3. A little bit about Udemy
We are a marketplace for online courses.
Instructors Students

4. Udemy’s Growth

5. Let’s talk about how to build a successful
experimentation program.

6. "To invent you have to
experiment, and if you
know in advance that it's
going to work, it's not an
experiment.”
- Jeff Bezos
Image source: huffpost.com

7. - Limited resources
- Few experiments per quarter
- No clear process
- Very little revenue contribution
coming from experiments
Before

8. - Limited resources
- Few experiments per quarter
- No clear process
- Very little revenue contribution
coming from experiments
Before
- 4 agile teams
- 40 - 50 experiments per quarter
- Established process
- Significant revenue contribution
Today

9. Our Approach
Step 1: Set Clear Business Metrics

10. Our Approach
Step 1: Set Clear Business Metrics
Step 2: Ideation

11. Our Approach
Step 1: Set Clear Business Metrics
Step 2: Ideation
Step 3: Prioritization

12. Our Approach
Step 1: Set Clear Business Metrics
Step 2: Ideation
Step 3: Prioritization
Step 4: Execution

13. 0

14. This is how I felt
Image source: beyondanxietyanddepression.com

15. How did we actually get there?
Step 1: Build the Right Process
Step 2: Build the Right Team
Step 3: Create the Right Culture

16. Step 1: Build the Right Process
Image source: 123rf.com

17. Process: The Experimentation Loop
Prioritize User
Problems
Form
Hypotheses
Gather Data
Prioritize
Hypotheses
Design
Build
Learn &
Improve
Test

18. Process: The Experimentation Loop
Prioritize User
Problems
Form
Hypotheses
Gather Data
Prioritize
Hypotheses
Design
Build
Learn &
Improve
Test

19. Gather Data
Rely on quantitative data to answer these questions:
- How do people use your product?
- Where do they get stuck?

20. How Do People Use Your Product: Pathway
Analysis

21. Pathway Analysis
Traffic Channel Landing Page Discovery Conversion

22. Pathway Analysis
Traffic Channel Landing Page Discovery Conversion
Note: Does not represent real Udemy numbers
30% Direct
40% SEO
20% Paid Acq

23. Pathway Analysis
Traffic Channel Landing Page Discovery Conversion
Note: Does not represent real Udemy numbers
30% Direct
40% SEO
20% Paid Acq
70% Product Page
30% Home Page

24. Pathway Analysis
Traffic Channel Landing Page Discovery Conversion
Note: Does not represent real Udemy numbers
30% Direct
40% SEO
20% Paid Acq
70% Product Page
30% Home Page
20% Search
10% Browse
2% Success Page

25. Pathway Analysis
Traffic Channel Landing Page Discovery Conversion
30% Direct
40% SEO
20% Paid Acq
70% Product Page
30% Home Page
1% Success Page20% Search
10% Browse
2% Success Page
Note: Does not represent real Udemy numbers

26. How Do People Use Your Product: Funnel Analysis
Understand the leakiest parts of your funnel
Product Landing Page
1,000,000
Watched the Preview Video
600,000
Created an
Account
60,000
Purchased
1,200
60%
10%
2%
Note: Does not represent real Udemy numbers

27. User
Segmentation
Matters
Image source: shortcourses.angliss.edu.au

28. 3 Questions to Ask to Figure Out the Right Segment

29. 3 Questions to Ask to Figure Out the Right Segment
1) Which type of users might be using your product differently?

30. 3 Questions to Ask to Figure Out the Right Segment
1) Which type of users might be using your product differently?
2) Which one of these users actually behave differently in your product?

31. 3 Questions to Ask to Figure Out the Right Segment
1) Which type of users might be using your product differently?
2) Which one of these users actually behave differently in your product?
3) Is the segment that’s behaving differently big enough for you to care?

32. 3 Questions to Ask to Figure Out the Right Segment
1) Which type of users might be using your product differently?
2) Which one of these users actually behave differently in your product?
3) Is the segment that’s behaving differently big enough for you to care?
For Udemy
We ran the pathway analysis and funnel analysis for these user segments:
- First time Udemy visitors vs returning visitors
- Mobile users vs desktop users
- Users coming from different traffic channels
- Users coming to buy different category of courses

33. We decided to focus on
First time Udemy visitors that landed on one of our product pages and didn’t take
any action on the page.

34. Process: The Experimentation Loop
Prioritize User
Problems
Form
Hypotheses
Gather Data
Prioritize
Hypotheses
Design
Build
Learn &
Improve
Test

35. Quantitative data is helpful
to understand how users behave

36. Quantitative data is helpful
to understand how users behave
but it’s not very helpful to understand
“why”

37. When users showed exit intent,
we asked a simple question:
What is preventing you from
purchasing this course right
now?
Qualitative User Research
Image source: blog.informizely.com

38. Prioritization of User Problems
1) Start with an open ended survey question to your users.
E.g - “What is preventing you from purchasing this course right now?”

39. Prioritization of User Problems
1) Start with an open ended survey question to your users.
E.g - “What is preventing you from purchasing this course right now?”
2) Bucket the answers in meaningful categories and this time ask as a
multi-choice question.
E.g “I will not purchase because the price is not right.” or “I don’t trust
Udemy yet”.

40. Top 3 Reasons For Udemy Were
- I don’t know if Udemy is the right place to learn this skill.
- I don’t understand how this course compares to the other courses.
- I don’t know if this course fits my skill level.

41. Now, we are so much more focused
The new focus:
“We would like to increase
Revenue Per Visitor for first
time visitors who land on
one of our product pages.
And we will do that by
solving these 3 problems for
users”
The old focus:
“We would like to increase
the Revenue Per Visitor.”

42. Process: The Experimentation Loop
Prioritize User
Problems
Form
Hypotheses
Gather Data
Prioritize
Hypotheses
Design
Build
Learn &
Improve
Test

43. Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
Form Hypotheses

44. 1) What is the problem you are solving?
Form Hypotheses
Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:

45. 1) What is the problem you are solving?
2) Which user segments have this problem?
Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
Form Hypotheses

46. 1) What is the problem you are solving?
2) Which user segments have this problem?
3) Is this a real problem? What evidence do we have to support that?
Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
Form Hypotheses

47. 1) What is the problem you are solving?
2) Which user segments have this problem?
3) Is this a real problem? What evidence do we have to support that?
4) How big is the problem?
Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
Form Hypotheses

48. Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
Form Hypotheses
1) What is the problem you are solving?
2) Which user segments have this problem?
3) Is this a real problem? What evidence do we have to support that?
4) How big is the problem?
5) How are we going to solve the problem

49. Form a clear hypothesis that ties your idea to the problem you are trying to solve.
Your hypothesis statement should answer these questions:
Form Hypotheses
1) What is the problem you are solving?
2) Which user segments have this problem?
3) Is this a real problem? What evidence do we have to support that?
4) How big is the problem?
5) How are we going to solve the problem
6) How will we know if this experiment succeeds?

50. Hypotheses Template
Based on user research, we know that [X] is an important user problem. If we do
[Y], we expect to see [this] change. This change should cause an increase in
[KPIs].

51. Hypotheses Template
Based on user research, we know that [X] is an important user problem. If we do
[Y], we expect to see [this] change. This change should cause an increase in
[KPIs].
Example: “Based on user research, we know that 40% of first-time visitors
are not sure if Udemy is the right place to learn a new skill. If we feature
successful student stories on our product pages, we expect to increase the
trust of our first-time visitors. This should cause lower bounce rates and
increased conversion rates."

52. Process: The Experimentation Loop
Prioritize User
Problems
Form
Hypotheses
Gather Data
Prioritize
Hypotheses
Design
Build
Learn &
Improve
Test

53. Prioritization of Hypotheses
1) ICE Model
2) Build a Customized Criteria for Your Own Business

54. ICE
Impact
Confidence
Ease

55. Customized Framework
Above the Fold
(Y/N)
Addressing an
issue discovered
via user
research?
(Y/N)
Did we have
success in a
similar experiment
before?
(Y/N)
Confidence
(1-5)
Ease of Building
(1-5)
Total Score
1 1 1 5 5 20
0 1 1 4 3 11
1 0 0 3 5 10

56. Process: The Experimentation Loop
Prioritize User
Problems
Form
Hypotheses
Gather Data
Prioritize
Hypotheses
Design
Build
Learn &
Improve
Test

57. Process: The Experimentation Loop
Prioritize User
Problems
Form
Hypotheses
Gather Data
Prioritize
Hypotheses
Design
Build
Learn &
Improve
Test

58. Analyzing the Results
Two key questions to answer:
1) Did the experiment solve the user problem?
2) Did solving the user problem move the needle?

59. Record and share your learnings and
decide on the next steps

60. Your Success Depends On:
1) How strong your hypotheses are

61. 1) How strong your hypotheses are
2) How strong your prioritization framework is
Your Success Depends On:

62. Your Success Depends On:
1) How strong your hypotheses are
2) How strong your prioritization framework is
3) How fast you can move through the cycle

63. Your Success Depends On:
1) How strong your hypotheses are
2) How strong your prioritization framework is
3) How fast you can move through the cycle
4) How much you are learning in each cycle

64. Step 2: Building the Right Team
Image source: www.scrumalliance.org

65. The Roles on the Team
- The Experiment Owner (Product Manager, Marketer, CRO Expert)
- Engineers
- User Experience Designer
- Data Analyst
- Data Scientist
- User Researcher

66. Managing Dependencies
Image source: agiletrainings.eu

67. Image source: agiletrainings.eu

68. To remove
dependencies,
create
dedicated,
cross-functional
teams with clear
decision makers.
Image source: www.solstice.com

69. Qualities of a Good Experimenter

70. Curious
Image source: www.icaneducation.ca

71. Analytical
Image source: www.eie.org

72. Comfortable
with
Uncertainty
Image source: www.digitaltrends.com

73. Detail
Oriented
Image source: www.mercerislandlearninglab.org

74. Strong
Leader
Image source: brookridgedayschool.com

75. Step 3: Create the Right Culture
Image source: www.halogensoftware.com

77. 8 Out of 10 Experiments Fail

78. Eliminate Fear of Failure
Image source: saraohara.com

79. Experiment to Win

80. Experiment to Win Learn

81. Reward not only wins, but also failed
experiments with important learnings

82. It Has Been a Winding Road
Image source: www.feco.info

83. This Is How We Started
Image source: www.fafsfoundation.org

84. This Is How We Felt Along The Way
Image source: www.sportpsychologytoday.com

85. This Is How We Are Feeling Now
Image source: www.ntd.tv

86. Image source: www.mannersmentor.com