Basics about A/B Testing

1. A/B Testing
By: Kapil Saxena

2. What Is A/B Testing?
 Develop
two versions of a page
 Randomly show different versions to users
 Track how users perform
 Evaluate (that's where statistics comes in)
 Use the better version

3. Why A/B Test?
A
typical website converts 2% of visitors
into customers
 People can't explain why they left
 Small changes can make a big difference
 How about +40%?
 Google believes it works, see Content
Experiments in Google Analytics

4. What Can You A/B Test?
 Removing
form fields
 Adding relevant form fields
 Marketing landing pages
 Different explanations
 Having interstitial pages
 Email content
 Any casual decisions you care about

5. A/B Tests Do Not Substitute For
 Talking
to users
 Usability tests
 Acceptance tests
 Unit tests
 Thinking

6. The G-Test
A
method for comparing 2 data sets
 It was invented by Karl Pearson in 1900
 It is a close relative of the chi-square test
 It is our main method for evaluating A/B
tests
 There are alternatives

7. Limitations Of The G-Test
 Only
answers yes/no questions (but you
pick the question)
 Only handles 2 versions (there is a
workaround)
 Requires independence in samples
 Does not do confidence intervals

8. What To Measure
 Start
your A/B test
 Divide your users into groups A and B
 Decide whether each person did what
you want
 Reduce your results to 4 numbers: ($a_yes,
$a_no, $b_yes, $b_no)

9. G-Test Evaluation
 Select
a yes/no question about users
 Divide users in A and B into yes/no
 Perform complicated G-test calculation
to calculate $p
 Our confidence is 1-$p
 Make decision if our confidence is near
100% and we have enough samples
 Enough samples is at least 10 yes and no
results in each test

10. The Conversion Funnel

11. Your Conversion Funnel






Every company has one or more conversion
funnels
You should know yours, and be actively trying
to improve each step
Each step can be tracked with some metric
Most A/B tests concentrate on one step in the
funnel
Expect to run multiple A/B tests against each
Standardize these metrics

12. Examples Of Metrics
 Sessions,
sessions with registration
 People who searched, who viewed detail
page, contacted, leased
 People who saved favorites, started a
cart, completed purchase
 People who saw at least 3 pages, clicked
on an ad
 Anything measurable and important to
your business

13. Too Many Metrics?
 You
may have many metrics
 High confidence on one may be chance
 Believe if it was the metric you tried to
change
 Believe if very high confidence
 Believe if metrics agree
 Conflicting metrics require business
decision

14. Is That It?
 You
now know enough to run a successful
A/B test!
 If you do everything right
 If you do it wrong you won't know
 You'll just get random answers
 And believe them

15. Compare Apples To Apples
 Traffic
behaves differently at different
times
 Friday night ≠ Monday morning
 First week in month ≠ last week in month
 Last week's visitors have done more than
this week's
 Do not try to compare people from
different times

16. Be Careful When Changing
The Mix
A
and B can receive unequal traffic
 But do not change the mix they get
 Wrong Change(90/10) A vs B to (80/20)
 You are implicitly comparing old A's with
new B's
 Right Change(10/10/80) A vs B vs
Untested to (20/20/60)
 This comes up repeatedly

17. What Is Wrong With This?
 Suppose
you are A/B testing a change to
your product page
 You log hits on your product page
 You log clicks on Buy Now
 You plug those numbers into the A/B
calculator
 Is this OK?

18. Beware Hidden Correlations!
 Correlations
increase variability, and
therefore $g_test
 Some people look at many product
pages
 Their buying behaviour is correlated on
those pages
 This increases the size of chance
fluctuations
 Leading to wrong results

19. Guarantee Independence
 Whatever
granularity (session, person,
event) you make A/B decisons on...
 Needs to be what you test on
 In this case measure people who hit your
product page
 Measure people who clicked on Buy Now
 Those are the right statistics to use
 This comes up repeatedly

20. Wrong Metric






At Rent.com we changed the title of our
lease report email
The new email had improved opens and
clicks
That was because it interested people who
were still looking for a place to live
That email needed to interest people who
had already found a place to live
We looked at the wrong metric, and it cost us
millions
This mistake is fairly rare

21. That's It!
 Those
are the big mistakes that I've seen
 You now know how to do an A/B test
 ...and should have good odds of getting it
right
 Of course there is more to know
 But this is the core

22. Thanks!!!