This is my presentation from June's Shame On UX, where we focused on split testing using real conversion data and confidence intervals to evaluate test results.
2. Split Testing 101
KISS
If you can’t measure it, you can’t test it
For clean data you can only test ONE
element at a time
Attribution is everything
Follow @ShameOnUX
3. Types of Conversions
• Purchase
• Contact
• Phone Call
• Sign-up
• Subscription (on-going purchase)
• Content
• Share
• Time on site / page
• Pages per visit
• Download
Follow @ShameOnUX
4. A Quick Note…
When testing for specific conversions, focus
on specific signals that drive the specific
conversion.
• Purchase vs. Sign up
• Submit content vs. Get the click
• Social Share vs. Link-bait
Follow @ShameOnUX
5. Split Test 1
Goal: Increase Content Acquisition
Conversion: Form Submittal
Type: user Generated Review
Motivation: Share frustration, publicize
experience, receive sentiment, receive
compensation
Follow @ShameOnUX
12. Variation 2 Test Results
Conversion Increase: 105%
Visitors Conversions Conversion Rate Standard Error
Control 15263 2106 13.80% 0.28%
Variation 14007 3957 28.25% 0.38%
95% Conversion Rate Limits
From To
Significant At 13.34% 14.26%
95% confidence: YES 27.62% 28.88%
99% confidence: YES
99% Conversion Rate Limits
Z-score -30.628883 From To
13.25% 14.35%
P-value 3E-206 27.50% 29.00%
Follow @ShameOnUX
13. Split Test 2
Goal: User Sign-Up
Conversion: Form Submittal
Type: Personal Information
Submission
Motivation: Interested in starting free
trial
Follow @ShameOnUX
23. Split Test 3
Scenario: Data supports that visitors who first visit the
product detail page prior to entering the checkout are
50% more likely to complete their purchase
Variation 1: Remove Buy Now Option
Element: Button
Follow @ShameOnUX
25. Split Test 3, Variation 1 Test Results
Conversion Decrease: 18%
Visitors Conversions Conversion Rate Standard Error
Control 10112 543 5.37% 0.22%
Variation 11213 511 4.56% 0.20%
95% Conversion Rate Limits
From To
Significant At 5.00% 5.74%
95% confidence: YES 4.23% 4.88%
99% confidence: YES
99% Conversion Rate Limits
Z-score 2.72335304 From To
4.93% 5.81%
P-value 0.99677 4.17% 4.94%
Follow @ShameOnUX
26. Split Testing Key Takeaways
• Don’t Guess. Test.
• If at First you don’t succeed, test again. If
at first you do succeed, test again.
• Remove emotion from testing
• Tests will fail
• Tests will succeed
• Put yourself in the shoes of your users; feel
their pain, now design a test
Follow @ShameOnUX