A quick presentation about A/B testing pitfalls you should be aware of. Things that influence your A/B tests and the results you get out them. I

1. A/B Testing Pitfalls
Michal Parizek (@parezem), Avast #MeasureCamp , March 2015

2. Low-hanging Fruit
(1) 2 business cycles
(2) Big enough data sample (minimum of 200 orders per testing
experience
(3) No bugs in A/B test setup
(4) Daily orders/revenue + cumulative orders/revenue
(5) Check http://abtestguide.com/calc/
@parezem | #MeasureCamp

3. Traffic Mix & Seasonality
Challenge
A/B test results are tight to the traffic
and circumstances of a testing
period
Solution
(1) Make sure all testing experiences get the same traffic mix.
(2) Avoid special commerce events for A/B testing (Christmas,
Black Friday, Valentine’s day etc.).
(3) In case you have a seasonal business, A/B test your hypotheses
in both on and off season.
3
@parezem | #MeasureCamp

4. Cross-device A/B Testing
Challenge
Attribution in cross-device A/B
testing. One user, different
devices, not the same testing
experience
Solution
(1) Use targeting only to one device type - not solve multiple same
device type issue
(2) Wait when tools add “user-centric testing”
@parezem | #MeasureCamp

5. Long Purchase Decision Making Process
Challenge
Customers from your A/B test made the actual decision
before your A/B test was launched
Solution
(1) Target new visitors only.
(2) Set micro-conversion goals when an A/B test focuses on an
early part of the purchase process
First visit
on the LP
T = 0 T + 28
Purchase!Third visit
on the LP
T + 12
Your A/B is
launched
T + 20
Research
Finances, laziness, more research,
discount coupons
@parezem | #MeasureCamp

6. Research Online, Purchase Offline
Challenge
Your online A/B tests influence
offline purchases too
Solution
(1) Show discount coupons for offline purchases - each testing
experience has unique discount coupon.
Experience A = DSC08A
Experience B = DSC08B
@parezem | #MeasureCamp

7. Optimising for Maximum CLTV
Challenge
You don’t want to get more average
customers. You do want to get more
excellent customers!
Solution
(1) A survey after a purchase - “Would you recommend us to your friend?”
(2) Re-evaluate your A/B tests after few months.
(3) Patterns with your historical data - predict CLTV.
(4) Immediate insights: e.g. a share of auto-renewal customers
buy again
recommend
stay loyal
upgrade
@parezem | #MeasureCamp