Individual tests drive insights & ROI, but the most sophisticated optimizers look beyond what an individual test is telling them and use data to optimize their overall testing performance.
In this talk, Claire will dive into the specifics of how to track, improve, and drive insight from performance metrics for your conversion program, so you can not only run better tests, but get more out of your investment in CRO.
15. Claire Vo @clairevo
Instead you should know:
What to know Metric to Measure
How many tests can I run Testing Capacity
How many am I running? Testing Velocity & Coverage
Am I getting any better? Trends Over Time
When Iâm not running tests, why? Annotations
16. Claire Vo @clairevo
Testing Capacity
CRO PERFORMANCE METRICS
How many tests can you run per year
52
Test Duration (in weeks)
x # of simultaneously
testable pages/funnels
17. Claire Vo @clairevo
Testing Velocity
CRO PERFORMANCE METRICS
How many experiments are run per [time period]
0
2.5
5
7.5
10
5/23/2015 6/20/2015 7/18/2015 8/15/2015 9/12/2015 10/10/2015 11/7/2015 12/5/2015
âą Weekly for high traffic sites
âą Monthly for low traffic sitesGoal 8 tests / week
18. Claire Vo @clairevo
Testing Velocity
CRO PERFORMANCE METRICS
How many experiments are run per [time period]
-0.4
2.2
4.8
7.4
10
5/23/2015 6/20/2015 7/18/2015 8/15/2015 9/12/2015 10/10/2015 11/7/2015 12/5/2015
âą Weekly for high traffic sites
âą Monthly for low traffic sitesGoal 8 / week
Tracking trend towards goal
BONUS
20. Claire Vo @clairevo
Ćukasz Twardowski
A/B Testing and the Infinite Monkey Theory
http://www.slideshare.net/useitbetter/ab-testing-and-the-infinite-monkey-theory
21. 13%
87%
Claire Vo @clairevo
Testing Coverage
CRO PERFORMANCE METRICS
What % of testable days are you running a test?
July
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31
August
1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31
83%
22. Claire Vo @clairevo
Time Since Last
Zero Test Day
CRO PERFORMANCE METRICS
It has been
100
DAYS SINCE WE HAD
NO TESTS LIVE
24. EXPERIMENT ENGINEEEClaire Vo @clairevo
Stop evaluating the âš
quality of your program âš
on a test-by-test basis.
(it makes you look good, until you look bad)
25. Claire Vo @clairevo
You should also track:
What to know Metric to Measure
Am I running effective tests?
Win Rate, Lift Amount, âš
Expected Value
Am I running tests effectively? ROI
Am I getting any better? Trends Over Time
27. Claire Vo @clairevo
Rolling Win Rate
CRO PERFORMANCE METRICS
What % of tests run win / lose / are inconclusive over time?
0
4
8
12
16
April May June July August September October November
Goal 15% Win Rate
Tracking by Page
BONUS
Once you hit a steady testing
velocity, ideally your win rate
stays level or increases.
28. Claire Vo @clairevo
Expected Value
CRO PERFORMANCE METRICS
$1 per roll of the die
Every time you roll a 3
I pay you $5
Win Rate = 1/6 = 16%
Value of Win = $5
Expected Value of Roll = 83 cents
DO NOT PLAY THIS GAME!
29. Claire Vo @clairevo
Expected Value
CRO PERFORMANCE METRICS
On average, what is the expected value of any test run?
Win Rate
x
Average Lift of Winning Test
x
Revenue Value of Test
30. Claire Vo @clairevo
Expected Value
CRO PERFORMANCE METRICS
$1500 every time you run a test
Every time you win, you get 10% increase
on your revenue, which is $1,000,000
and you win 10% of the time
Win Rate = 10%
Value of Win = 10% x $1,000,000 = $100,000
Expected Value of Test = $10,000
RUN THIS TEST FOR SURE!
31. Claire Vo @clairevo
Annual Expected Value of Program
CRO PERFORMANCE METRICS
Expected Value of Test
x
Annual Testing Capacity
What is the estimated potential of my testing program?
32. Claire Vo @clairevo
Annual Expected Value of Program
CRO PERFORMANCE METRICS
What is the estimated potential of my testing program?
High Complexity
Low Velocity
Low Complexityâš
High Velocity
Type of Tests Take longer, bigger changes Easier, smaller changes
Win Rate 20% 10%
Avg Lift 30% 10%
Expected Value of Test $30,000 $10,000
Tests / Year 25 100
Annual Expected Value âš
of Testing Program $1.5M $1M
34. Putting it all together
EXPERIMENT ENGINEEEClaire Vo @clairevo
35. CRO Metrics for Performance:
EXPERIMENT ENGINEEEClaire Vo @clairevo
1. Track quantity by measuring velocity & coverage
2. Track quality by win rate, avg. lift, and expected value
3. Set goals & measure everything over time
36. EXPERIMENT ENGINEEEClaire Vo @clairevo
Questions?
claire@experimentengine.com
experimentengine.com/cxl-live-2016