This document summarizes a presentation on data-driven UI/UX design and A/B testing. It discusses how Obama raised $60 million through simple experiments on his website, testing different button text and media campaigns. The presentation outlines a 4-step process for A/B testing: 1) using data to identify problems, 2) forming hypotheses, 3) running experiments, and 4) evaluating results and deciding next steps. It emphasizes establishing key metrics and sample sizes to determine statistically significant results. Iteration is important - new hypotheses should be based on learnings. Regular testing allows continuous improvement even with small changes.
5. Obama raised $60 Million by Running a Simple Experiment.
Challenge: Raise money if no one has heard of you and you only have a
website?
Idea: Every visitor to the site is an opportunity to generate lead, convert
them into volunteer for donation
9. … And the winner is?
E-mail Sign up Rate
11.6%
+40.6%
Vs. Original 8.26%
10% of email subscribers convert into volunteers
21$ donation per volunteers on average
+2,880,000 e-mails
+60 millions in donation
10.
11. A scientific method to measure how a new idea
(version B/variation) performs against an
existing implementation (version A/control)
12. Where does this belong to?
● Conversion Rate Optimization (CRO)
● A/B Test: Best 1 for All
● Segmentation: Best 1 for Some
● Personalization: Best 1 for 1
Our focus for today!
14. HiPPO
Highest Paid Person’s Opinion
Highest Paid Person’s in the Office
Higher-paid or higher-rank employees have the biggest say when a decision has to be made.
Describe an organization's reliance on human instinct rather than data in the decision-making process.
15. “My design is much more beautiful and intuitive compared to what we
have now! It’s so obvious, don’t think we need to run an A/B test”
“My wife (of the boss) told me that this banner is confusing. Let’s just
remove it!”
“Let’s put our team focus on launching this feature. This is gonna be
huge!”
44. 4 Steps Framework of A/B Testing
Step 1
Data
Figure out what
to improve
Step 2
Hypothesis
Make an
educated guess
Step 3
Experiment
Test
your guess
Step 4
Act
Decide what’s
next
52. Find pages with HIGH traffic and HIGH bounce rate
● Landing pages in top 10-20
with bounce rate much higher
than site average
● This is where you are burning
your marketing dollars without
caring about the full user’
journey
53. Find pages with HIGH traffic but LOW page value
● These are the pages with highest traffic
going through
● But does not contribute much to
conversion action
● How to make it work for you?
54. Find the MOST painful areas on the buying journey
● Identify the steps with highest
drop-off rate to optimize
● Understand where users go to
next
55. Find the MOST interacted features with LOW Conversion Rate
57. What kind of qualitative data you need?
● User testing (video of how users browse your site)
● Survey data (for detailed feedbacks)
● Heat mapping (quick glance at elements interaction)
● Customer service
● ...
59. Use exit surveys to learn why people leave
Apply this survey on the funnel step with the highest drop-off
60. Use post-purchase survey to understand why people buy
Know what drive conversions to know your selling points, customer’s perception to drive
conversions for others
67. 1. Population Size
3 Key Criterias
2. Potential Performance Improvement
3. Estimated Technical Difficulty to Set up
You can assign a score for each of this and relevant weightages
71. What’s the aim of having a hypothesis?
The hypothesis should:
● Capture the essence of the change you propose to make and what you
think the effect will be
● A clear understanding and a plan that addresses what you will learn by
testing the hypothesis
72. Ask yourself these questions when crafting hypothesis
● If you fail, what did you learn that you will apply to future designs?
● If you succeed, what did you learn that you will apply to future
designs?
● How much work are you willing to put into your testing in order to
get this learning?
89. 2 Types of KPIs
Primary metrics: The key metrics that will be directly impacted if the
hypothesis hold true.
Example: If you A/B test a clearer CTA message, then: KPI = CTR on this
CTA
Secondary metrics: The indirect but important metrics that would be resulted if
the hypothesis hold true.
Example: If you A/B test a clearer CTA message, then: KPI = CR (as an users
would be more likely to checkout due to clearer messaging)
90. Good vs Bad KPIs
Bad KPIs:
● No. of pageviews
● Time spend on pages
● No. of sign-ups
● No. of transactions
● ...
Good KPIs:
● Share of page views with
at least 75% scroll
● Share of page views with
at least 10 mins. on page
● Sign-up Rate
● Conversion Rate
● ...
2 common problems:
● Vanity metrics
● Not an effectiveness metrics
95. Confidence Interval (CI)
P = Conversion Rate
Z will be calculated based on significance level
The more confidence you want to be that the
difference is real → The bigger statistical
significance → The bigger z will be
Hence, if you want to have a more confident
range of the actual CR, the bigger sample size
will be required
97. Launch or keep testing?
1.3%
1.6%
1.7%
2.5%
Control
Var A
Var B
Var C
Conversion Rate
Confidence Interval
98. If you not only want to be certain of the change but also of the magnitude of the change
Minimum Detectable Effect (MDE)
The minimum relative change in conversion rate you would like to be able to detect - MDE
You’ll need less traffic to detect big changes and more traffic to detect small changes
To demonstrate, let’s use an example with a 20% baseline conversion rate and a 5% MDE.
Based on these values, your experiment will be able to detect 80% of the time when a variation
underlying conversion rate is actually 19% or 21% (20%, +/- 5% × 20%)
99. The smaller the improvement
impacts that you want to
detect (MDE), the bigger the
required sample size to be
really sure that you can detect
that small change when it
happens - MORE
OPPORTUNITY
When to stop the test to evaluate?
The more significant you want
to be (as less chance of being
wrong as possible), the
bigger the required sample
size - SAFER
100. Always define the sample size before running the test
● Should not peek the result before you hit the deadline
● Should not end the experiment early before the deadline even when you hit stat. significance
● Should not extend the test past the deadline hoping to reach significance
107. 4 Steps Framework of A/B Testing
Step 1
Data
Figure out what
to improve
Step 2
Hypothesis
Make an
educated guess
Step 3
Experiment
Test
your guess
Step 4
Act
Decide what’s
next
120. If you are not testing, you are not learning
Always have test running, no matter how small
If you don’t have a process, you won’t
continuously improve