Ruben shows a step-by-step method of using research and experimentation to combine all learnings into behavioral insights. With this method, you will not only improve your research and conversion rate optimization practices but actually learn and provide a much more pleasant journey for your potential customers! This will undoubtedly help you increase conversion rates and become a lot more successful in your job.

1. Meta-analyses in Experimentation:
The whats and hows
Webinar VWO
Ruben de Boer | ruben.de.boer@onlinedialogue.com

2. Brief introduction
Ruben de Boer
https://www.linkedin.com/in/rgdeboer/

3. Who experiments?

4. Who documents experiment
learnings?

5. Who combines learnings from
multiple A/B tests to learn
about customer behavior?

6. Goal of this presentation
With an easy tweak, truly learn much more while also
heavily decreasing your biases.
The goal is to use meta-analyses to know what
hypothesis to address in which step of the customer
journey and how.
This will help you become much more successful.

7. 25%

8. 75%

10. Conversion Rate Optimization

11. Conversion Rate Optimization

12. Conversion Rate Optimization

13. Conversion Rate Optimization
As seen in: Experimentation Works (p. 16) by Stefan Thomke

14. 100% LEARN

15. Hierarchy of Evidence
Meta analysis

16. Meta-analysis
● An analysis that combines the results of multiple studies
● A single study (A/B test) can be prone to errors
● The meta-analysis aims to derive a pooled estimate closest to the truth

17. Meta-analysis
A B

18. Meta-analysis
A B

19. Hierarchy of Evidence
Meta analysis

20. 1. Create Behavioral Hypotheses based
on your research

21. Behavioral hypotheses
● General hypotheses stating something about your visitors’ behavior, needs,
and motivations
● Based on your user, data and scientific research & completed experiments
● Cluster insights that belong together
1 2 3 4 5

22. Behavioral hypotheses
1 2 3 4 5

23. Behavioral hypotheses
● People buy Google merchandise because they
love the brand (several sources)
● People want to be part of the Google community
(several sources)
● People identify themselves with a brand (science)
● Customers are a fan of Google (poll)
● Customers use many Google products
(interviews)
1 2 3 4 5

24. Behavioral hypotheses
● People buy Google merchandise because they
love the brand (several sources)
● People want to be part of the Google community
(several sources)
● People identify themselves with a brand (science)
● Customers are a fan of Google (poll)
● Customers use many Google products
(interviews)
By elaborating on the brand and
community feelings, sales increase
1 2 3 4 5

25. Behavioral hypotheses
● Test 1: Value proposition on landing page
● Test 2: Display the number of Google fans
world-wide
● Test 3: Display pictures of a large Google event
● Test 4: Elaborate on being the official Google
merch store
● Etc…
By elaborating on the brand
and community feelings,
sales increase
AB test AB test AB test
AB test AB test AB test
AB test AB test AB test
1 2 3 4 5

26. Behavioral hypotheses
Aim for 5-10 behavioral hypotheses
Customer problem: Visitors have a hard time finding the right products.
Behavioral hypothesis: By making it easier for the visitor to find the right products, sales increase.
Customer problem: Visitors require social proof and feel the need to belong.
Behavioral hypothesis: By increasing social proof, sales increase.
1 2 3 4 5

27. Behavioral hypotheses
Aim for 5-10 behavioral hypotheses
Customer problem: Visitors have a hard time choosing the right product.
Behavioral hypothesis: When including guidance and advice on the right product, sales increase.
Customer problem: Visitors are hesitant to purchase due to feelings of uncertainty regarding the product,
delivery, and terms.
Behavioral hypothesis: By providing certainty, sales increase.
1 2 3 4 5

28. 2. For every experiment document the
page, behavioral hypothesis, and
optimization strategy

29. Ability Attention Motivation Certainty Choice architecture
Optimization strategies
The five most important ways to optimize your journey based on psychological knowledge
1 2 3 4 5

30. Ability
You want to cross the bridge, but the bridge is broken
1
0
t
i
p
s

31. Attention
There are two bridges in front of you. A normal one and the other made of gold.
Which one will you cross?

32. Motivation
You could cross the bridge, but why would you?

33. Certainty
You want to cross the bridge, but only if you trust the one who built it.

34. Choice
Architecture
There are two prices for crossing the bridge:
- Normal ticket €5
- Premium ticket €8.50 <- Most chosen

35. 1 2 3 4 5

36. https://www.udemy.com/course/cro-process/
1 2 3 4 5

37. 1 2 3 4 5

38. By elaborating the brand and community
feelings, sales increase
1 2 3 4 5

39. By elaborating the brand and community
feelings, sales increase
AB test AB test AB test
AB test AB test AB test
AB test
HOME PAGE
1 2 3 4 5

40. By elaborating the brand and community
feelings, sales increase
AB test AB test AB test
AB test AB test AB test
AB test AB test
LIST PAGE
AB test AB test AB test
AB test AB test AB test
AB test
HOME PAGE
1 2 3 4 5

41. By elaborating the brand and community
feelings, sales increase
AB test AB test AB test
AB test AB test AB test
AB test AB test
LIST PAGE
AB test AB test AB test
AB test AB test AB test
AB test
HOME PAGE
AB test AB test AB test
AB test AB test
PRODUCT PAGE
1 2 3 4 5

42. By elaborating the brand and community
feelings, sales increase
Motivation Ability Motivation
Motivation Motivation Attention
Certainty
Choice
architect.
LIST PAGE
Ability Motivation Motivation
Certainty Motivation Attention
Choice
architect.
HOME PAGE
Motivation Motivation Certainty
Ability Attention
PRODUCT PAGE
1 2 3 4 5

43. By elaborating the brand and community
feelings, sales increase
Motivation Ability Motivation
Motivation Motivation Attention
Certainty
Choice
architect.
LIST PAGE
Ability Motivation Motivation
Certainty Motivation Attention
Choice
architect.
HOME PAGE
Motivation Motivation Certainty
Ability Attention
PRODUCT PAGE
Winning
A/B test
Losing
A/B test
Inconcl.
A/B test
1 2 3 4 5

44. 1 2 3 4 5

45. 3. Set up meta-analyses in your
documentation tool

46. Meta analyses
Page - Behavioral hypothesis
HOME PAGE
Completed experiments: 130
Behavioral hypothesis Completed exp. Win-rate Avg. uplift per winner
Brand feeling 40 25% 4%
Find right products 35 20% 6%
Social proof 20 70% 8%
Guidance and advice 25 30% 1%
Certainty 10 0% -
1 2 3 4 5

47. Meta analyses
Page - Behavioral hypothesis
HOME PAGE
Completed experiments: 130
Behavioral hypothesis Completed exp. Win-rate Avg. uplift per winner
Brand feeling 40 25% 4%
Find right products 35 20% 6%
Social proof 20 70% 8%
Guidance and advice 25 30% 1%
Certainty 10 0% -
1 2 3 4 5

48. Meta analyses
Page - Behavioral hypothesis
CHECKOUT
Completed experiments: 80
Behavioral hypothesis Completed exp. Win-rate Avg. uplift per winner
Brand feeling 20 60% 7%
Find right products 15 20% 5%
Social proof 10 5% 2%
Guidance and advice 25 30% 2%
Certainty 10 0% -
1 2 3 4 5

49. Meta analyses
Strategy - page
ABILITY
Completed experiments: 80
Page Completed exp. Win-rate Avg. uplift per winner
Home page 12 30% 3%
Category page 15 20% 2%
Product page 25 75% 11%
Cart 18 10% 9%
Checkout 10 0%
1 2 3 4 5

50. Meta analyses
Strategy - Behavioral hypothesis
MOTIVATION
Completed experiments: 85
Behavioral hypothesis Completed exp. Win-rate Avg. uplift per winner
Brand feeling 25 70% 6%
Find right products 10 22% 5%
Social proof 15 55% 3%
Guidance and advice 20 20% 4%
Certainty 15 0%
1 2 3 4 5

51. 1 2 3 4 5

52. Hierarchy of Evidence
Meta analysis
1 2 3 4 5

53. A/B test report
● Name of the A/B test
● Reason for the test & hypothesis
● Setup & screenshots
● Results on main KPIs
● A/B test learnings
● Conclusions & recommendations
1 2 3 4 5

54. A/B test learnings
● Overall the variant resulted in a 4.3% uplift
● The hypothesis is confirmed for mobile users
● Results on desktop were inconclusive
● Especially new users seem to like the change
● For users coming from a paid marketing campaign, the uplift was highest
(5.1%)
1 2 3 4 5

55. A/B test learnings
Use your data and meta-analyses
● Was the hypothesis confirmed? If not, was the hypothesis wrong or the execution?
● Combine the results of this experiment with what you already know from
meta-analyses on that page, on the hypothesis and on the strategy
● What could these insights say about your customers' needs, motivations, and
behavior?
● With this knowledge, is there anything you would change in your approach?
Different hypothesis / strategy / page?
● What are good follow-up experiments?
1 2 3 4 5

56. WHICH hypothesis
WHAT page
HOW (optimization strategy)
Now you know:

57. 4. Set up evidence-based prioritization

58. Prioritization
HOME PAGE
Completed experiments: 130
Behavioral hypothesis Completed exp. Win-rate Avg. uplift per winner
Brand feeling 40 25% 4%
Find right products 35 20% 6%
Social proof 20 70% 8%
Guidance and advice 25 30% 1%
Certainty 10 0% -
1 2 3 4 5

59. Prioritization
HOME PAGE
Completed experiments: 130
Behavioral hypothesis Completed exp. Win-rate Avg. uplift per winner
Brand feeling 40 25% 4%
Find right products 35 20% 6%
Social proof 20 70% 8%
Guidance and advice 25 30% 1%
Certainty 10 0% -
1 2 3 4 5

60. Prioritization
Expected impact of test
win-rate * avg. uplift per winner
Completed exp. Win-rate Avg. uplift per winner
Home page
Social proof
20 70% 8%
1 2 3 4 5

61. Prioritization
Expected impact of test
win-rate * avg. uplift per winner
Test idea related to social proof on the home page
70%*8% = 5.6
Completed exp. Win-rate Avg. uplift per winner
Home page
Social proof
20 70% 8%
1 2 3 4 5

62. Meta-analysis feedback loop
WINNER
Home page
Social proof
WIN-RATE PRIO. SCORE
KEEP TESTING
Home page
Social proof
1 2 3 4 5

63. Meta-analysis feedback loop
WINNER
Home page
Social proof
WIN-RATE PRIO. SCORE
KEEP TESTING
Home page
Social proof
1 2 3 4 5
WIN-RATE PRIO. SCORE
Inconclusive or
loser Test different
hypothesis

64. Prioritization
Add 1-3 additional attributes if you like
● Alignment with business goals and OKRs (important test goals get a higher score)
● Percentage of traffic that will see the change (above the fold gets a higher score)
● Minimum detectable effect (lower MDE gets a higher score)
● Revenue going through the page (higher percentage receives a higher score)
● Urgency (more urgent, means a higher score)
● Ease (make sure to balance easy and complex tests for velocity and impact)
1 2 3 4 5

65. Prioritization
Add 1-3 additional attributes if you like
● Alignment with business goals and OKRs (important test goals get a higher score)
● Percentage of traffic that will see the change (above the fold gets a higher score)
● Minimum detectable effect (lower MDE gets a higher score)
● Revenue going through the page (higher percentage receives a higher score)
● Urgency (more urgent, means a higher score)
● Ease (make sure to balance easy and complex tests for velocity and impact)
Ensure the evidence-based scores have the highest impact on the overall score
1 2 3 4 5

66. Prioritization
Add 1-3 additional attributes if you like
● Alignment with business goals and OKRs (important test goals get a higher score)
● Percentage of traffic that will see the change (above the fold gets a higher score)
● Minimum detectable effect (lower MDE gets a higher score)
● Revenue going through the page (higher percentage receives a higher score)
● Urgency (more urgent, means a higher score)
● Ease (make sure to balance easy and complex tests for velocity and impact)
Ensure the evidence-based scores have the highest impact on the overall score
(win-rate * avg. uplift per winner)*5 + Business goal score + MDE score + Ease
1 2 3 4 5

67. Prioritization
https://www.linkedin.com/in/rgdeboer/
1 2 3 4 5

68. 5 steps to truly get to know
your digital users
1) Create Behavioral Hypotheses based on your
research
By elaborating the brand
and community feelings,
sales increase

69. 5 steps to truly get to know
your digital users
1) Create Behavioral Hypotheses based on your
research
2) For every experiment, document the page,
behavioral hypothesis, and optimization strategy

70. 5 steps to truly get to know
your digital users
1) Create Behavioral Hypotheses based on your
research
2) For every experiment, document the page,
behavioral hypothesis, and optimization strategy
3) Set up meta-analyses in your documentation tool

71. 5 steps to truly get to know
your digital users
1) Create Behavioral Hypotheses based on your
research
2) For every experiment, document the page,
behavioral hypothesis, and optimization strategy
3) Set up meta-analyses in your documentation tool
4) Set up evidence-based prioritization

72. 5 steps to truly get to know
your digital users
1) Create Behavioral Hypotheses based on your
research
2) For every experiment, document the page,
behavioral hypothesis, and optimization strategy
3) Set up meta-analyses in your documentation tool
4) Set up evidence-based prioritization
5) Share your new insights with your colleagues and
celebrate :-)

73. Thank you!
Ruben de Boer
ruben@onlinedialogue.com
linkedin.com/in/rgdeboer/
Affordable bestselling CRO courses
https://conversionideas.com/online-courses