To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.
10. Asking the right questions is hard.
Arm yourself with data.
#ScienceOfTesting
11. #ScienceOfTesting
Use quantitative & qualitative data
Quantitative data
tells you
where to test
Qualitative data
gives you an idea of
what should be tested
14. Don’t choose tests randomly
Access this spreadsheet in this blog post: http://blog.optimizely.com/2014/07/02/
how-to-use-data-to-choose-your-next-ab-test/
17. #ScienceOfTesting
Parts of a hypothesis
“If [Variable], then [Result], because [Rationale].”
• The element that is modified
• Isolate one variable for an A/B test
• Call to action, visual media, forms
18. #ScienceOfTesting
Parts of a hypothesis
“If [Variable], then [Result], because [Rationale].”
• The predicted outcome
• Use data to determine the size of effect
• More email sign-ups, clicks on a CTA
19. #ScienceOfTesting
Parts of a hypothesis
“If [Variable], then [Result], because [Rationale].”
• Demonstrate your customer knowledge
• What assumption will be proven wrong if
the experiment is a draw or loses?
20. #ScienceOfTesting
All hypotheses are not created equal
Weak Hypothesis
If the call-to-action is shorter, the
conversion rate will increase.
Strong Hypothesis
If the call-to-action text is changed to
“Complete My Order,” the conversion
rates in the checkout will increase,
because the copy is more specific and
personalized.
21. #ScienceOfTesting
All hypotheses are not created equal
Weak Hypothesis
If the checkout funnel is shortened to
fewer pages, the checkout completion
rate will increase.
Strong Hypothesis
If the navigation is removed from
checkout pages, the conversion rate
on each step will increase because our
website analytics shows portions of
our traffic drop out of the funnel by
clicking on these links.
29. #ScienceOfTesting
What are we looking for?
• How confident am I that the observed difference
from my experiment was not due to chance?
• 95% Statistical Significance = 5% probability that
the observed difference was due to chance.
32. #ScienceOfTesting
Once you reach significance:
• Variation wins: Launch the variation or update
your website.
• Original wins: Learn why hypothesis was
incorrect.
• In either case: Think about what to test next.
38. #ScienceOfTesting
Step 2: Hypothesis
“If [Variable], then [Result], because [Rationale].”
If prospects’ access to a free trial is gated by a conversation with a sales rep, we’ll be able
to increase prospect to trial conversion rate.
Talking to sales will ensure all their questions get answered, improving their overall
experience and increasing willingness to take the next step with RJMetrics.
39. #ScienceOfTesting
Step 3: Experiment
• Changes to heading text
• Custom fields in Salesforce.com
• Business process changes for
sales reps
• Custom analysis in RJMetrics
based on offline conversion event
42. Marketing
Increase the
impact of your
tests by bringing
more team
members into the
process
#ScienceOfTesting
Product
Sales
Engineering
43. Document your test results in a central repository.
#ScienceOfTesting
Heat maps
Optimizely
results
Hypothesis
What we
learned
Variations
44. #ScienceOfTesting
Other tried and true tactics
• Build excitement by sharing your wins with the company
• Hold a competition for the biggest winning variation
• Votes on variations to see who has the highest accuracy
of predicting winners