Michele Kiss gave a presentation on how analytics can be used to optimize experiments and testing. She discussed how analytics can help define success metrics, generate hypotheses for testing, inform test ideas, and provide credibility for test results. Analytics can also help prioritize test ideas by estimating the potential revenue impact. Michele provided several examples of how companies have used analytics to improve site redesigns, signup flows, trials-to-paid conversions, and more.
28. @michelejkiss #CH2017
“I suppose it is tempting,
if the only tool you have is a
hammer, to treat everything
as if it were a nail."
Abraham Maslow, “The Psychology of Science” (1966)
33. @michelejkiss #CH2017
What sells your idea better?
“I believe that …[test idea]”
vs
“I believe that [test idea]…
and with a 2% increase in conversion, that
would drive a $1MM increase in revenue”
34. @michelejkiss #CH2017
{example} Revenue Forecast
mkiss.me/forecast-impact
• Use existing data
- Traffic, conversion, revenue
• Layer on assumptions
• Estimate impact
36. @michelejkiss #CH2017
{example} Revenue Forecast
• Current traffic
• Conversion to product pages
• % that view images
• Current conversion rate
• Current conversion rate for image-viewers
37. @michelejkiss #CH2017
{example} Revenue Forecast
Add your assumptions!
• What % of Product Page and/or Image
Viewers will view Video?
• What % lift in conversion will that drive?
• Any increase in AOV?
• Incremental revenue gain?
43. @michelejkiss #CH2017
Statistical significance is not enough
“Another way to think of it is that statistical significance isn’t enough.
Large sites … get thousands of conversions per hour, and can see
significance from their testing efforts quite quickly.
But this doesn’t mean they should only run their tests for a few hours.
Rather, they should run their tests long enough to capture a
representative sample of their users over time.”
- Peter Borden, Sum All
46. @michelejkiss #CH2017
Understand your seasonality
6.0%
6.5%
7.0%
7.5%
8.0%
8.5%
9.0%
9.5%
10.0%
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Year 1
Year 2
Year 3
Year 4
Year 5
86. @michelejkiss #CH2017
{example} Content Site
Web + Ad Analytics allowed us to value
every.single.page.
• How many (and which ads) are on that page
• How well sold the ads were
• What CPM we were charging
88. @michelejkiss #CH2017
Go forth and analyse!
ü Find opportunities
ü Estimate the impact
ü Dig in to your results
ü Take your testing further
ü Oh, and befriend your
analysts!