Optimizely - Stream:20 webinar of Dec. 4th 2014
In this webinar Alia Formoy, Sr. Digital Marketing Consultant at Stream:20, explains how to structure your tests and build a best in class testing program. She advocates a process of Exploration as opposed to Refinement.
Alia explains a 5 step process for effective optimisation:
1. Define Success
2. Identify Bottlenecks
3. Prioritise
4. Test
5. Optimise
She will dive into each of these steps and explain it with many examples.
For those of you who are struggling with formulating hypotheses, I recommend you to look at the part on Identifying Bottlenecks!
For a recording of the full webinar, go to: https://www.youtube.com/watch?v=SO-MI0rxiu0
For more info we recommend that you check out:
www.optimizely.com/gettingstarted/
www.stream20.com
1. Build a Best In Class
Testing Program - WEBINAR
Alia Formoy
2. Who are Stream:20?
Digital Sales
Planning
Digital
Organisation
Digital Sales
Toolkit
Digital Sales
Delivery
Optimising your business
structure and skillsets for
digital success
Ensuring the correct
toolkits are implemented
for success
Providing innovative
strategic support and
analysis to drive your
digital sales function
Driving incremental
revenue across
Desktop/ Mobile/Tablet
The Performance-Driven Digital Marketing Consultancy
We push up KPIs by rolling out good practice & best practice
on the client side
London Munich San Francisco
3. Who we work with
Retail Finance
Gaming
Telecoms
Travel Entertainment
Publishing
Technology
Utilities
4. This webinar – what we are going to learn today
Testing allows us to improve and optimise all our marketing channels and, importantly, the
website. By the end of this webinar, you will be able to:
• Build an effective Test Plan
• Roll out and refine
• Hit your targets for success
•Carrying out tests in
a planned and
strategic way.
1. Testing
•Understanding the
impact of tests and
the results they
produce.
2. Analysis •Running further tests
to validate and
improve results, and
incorporating
findings into BAU
3. Optimisation
6. The Testing Process
4. Test
• Run the test! Log
into Optimizely
and click "Create
New Experiment."
3. Prioritise
• Of the many things
you could test,
which one makes
the most sense to
start with?
2. Identify
bottlenecks
• How do users get
from landing on
your site to a
"success" state?
• Where do users
drop out or stop
moving along the
success path?
1. Define
success
• What metrics do
you use to
quantify success?
• What user actions
do you consider
"success"?
7. 1. Define success: Objectives
• First step should be the easiest: What do you want to improve?
• What are some of the possible objectives for testing?
• Align with overall business objectives
Business
Goals
• Hit Sales target
Marketing
Goals
• Convert more prospects
into customers
Optimisation
Goals
• Reduce drop out
from payment
journey
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
9. You should begin your planning with
looking at where to test.
Use your analytics to uncover the
following: Exit
Exit
Exit
Exit
2. Identify bottlenecks: Where do you test?
Exit
Customermovement
Entry to
Site
Product
Page
Shopping
Basket
Your
Information
Review
Order
Order
Completed
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
• Top landing pages
• Top exit pages
• Top highest bounce rate pages
• Top abandonment points in your
funnel
• Top most valuable pages to your
business
• Top converting pages
• Lowest converting pages
10. 2. Identify bottlenecks: Test Structure
• Observation: This is the problem you are having which keeps
you from success.
• Hypothesis: A way to overcome the problem and get to success.
• One of the keys to effective testing is to test a hypothesis – to
help us achieve an objective
• Using data and your experience / best-practice knowledge to
determine what aspects might make a difference in achieving
the objectives
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
11. 2. Identify bottlenecks: Shortcuts
Observation:
Customers are forced to go from:
Test idea:
Insert an ‘Add to Basket’ button to the
products listed.
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
Category
Product
detail
Basket
Hypothesis:
Allowing customers to skip the middle
step would increase conversion to sale
12. 2. Identify bottlenecks: Guidance
Observation: There is a lack of clarity of which fields are mandatory and what each field
means.
Hypothesis: Guiding people through the form process will help increase conversion.
Test idea: Use ‘tooltip’ image and
hover text to provide guidance as
you fill out the form, rather than
error messages after each
submission
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
13. 2. Identify bottlenecks: CTA
Test: Insert additional ‘Add to
basket’ button at bottom of
reviews section.
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
Observation: Clicking on
Product reviews pushes
you to the bottom of the
Product Page where
there is no CTA.
Hypothesis: Making it
easy for those reading
reviews to add to basket
would increase
conversion.
14. 2. Identify bottlenecks: High Consideration
Test: Push Webchat
functionality harder in
navigation and as a timed pop
up
Observation:
Analytics show prospects
are going to FAQs section
and not returning.
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
Hypothesis: Prospects
need more interactive help
during the quote process.
15. 2. Identify bottlenecks: Mobile KPIs
Test: Add a link to the subscription page in the header of the page.
Observation: These mobile sites are media focused but Time Inc’s KPIs are subscriptions
and mobile conversion is low
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
Hypothesis: Pushing a subscription CTA as a focus will drive mobile conversion up
16. 2. Identify bottlenecks: Social Trust
Observation: Not being aware of product quality and seller reputation is a barrier
to conversion.
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
Test: Build trust in the product and seller by using user generated reviews as
social proof.
Hypothesis: Reinforcing ways to show quality of product and seller will increase
conversion.
17. 2. Identify bottlenecks: Social Satisfaction
Observation: Buying wine online has a barrier of not knowing whether it will be good quality.
Test: Influence buying behaviour by using customer satisfaction as social proof.
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
Hypothesis: Highlighting likelihood of previous consumers of buying again would decrease the barrier
to conversion.
18. 2. Identify bottlenecks: Social Urgency
Observation: Conversion from searching for flights to sale can be quite low.
Test: Influence buying behaviour with urgency using live data from the website.
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
Hypothesis: Creating a sense of urgency will increase conversion.
19. 2. Identify bottlenecks: Device Targeting
Hypothesis: Using device specific calls to action may increase conversion.
Observation: The conversation to quote rate from the iPad audience was much
lower than from Desktop.
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
Test: Use iPad style ‘swipe button’ for the CTA
20. 2. Identify bottlenecks: Data Capture 4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
Observation: Betfair’s Newsletter sign ups have slowed down on website.
Hypothesis: Driving opt ins with an incentive may increase signs ups and conversion.
Test: Push sign ups using a hero
campaign and bonus code as an
incentive to bet.
22. 3. Prioritise: Benefits of the Hypothesis Process 4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
A good hypothesis
is crucial to CRO,
otherwise you just
test random stuff
blindly.
The hypotheses you
generate from
insight and research
will result in a list of
possible solutions.
The solutions will
form the basis of
your test plan.
23. 3. Prioritise: Building the Testing plan
Map out testing priorities based on:
• Business impact – how hard is it
to run this test?
• Performance impact – how much
could this test improve
performance?
Commit to what your
measure of success will be
Commit to when you
measure performance
Repeat tests to exclude any
false results and plan when
results will be rolled out and
become BAU
Work out what
assets are needed
to run this test
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
Your list of potential
solutions based on your
hypotheses.
25. 4. Test: Analysis made easy
• Optimizely will make it easier for you to know when test results are
statistically significant so you can clearly see which variation wins.
4. Test3. Prioritise
2. Identify
bottlenecks
1. Define
success
26. The Optimisation Process
Set Goals
Gather
Insights
Form
Hypothesis
Test
Analyse
Implement
Learnings
1. Define
Success
2. Identify
Bottlenecks
3.
Prioritise
4. Test
5. Optimise
28. The benefits of Analysis
Original:
Sell
Build Trust
+ Sell
Desktop
Mobile
Tablet
Sell
Sell+ build
trust email
CR: 2%
Even if a test doesn’t give you an immediate win, if it’s planned and analysed
right, you’ll learn something that can be fed into the next test round.
CR: 6%
CR: 7%
CR: 3%
CR: 5%
CR: 2%
CR: 3%
5.
Optimise
4. Test
3.
Prioritise
2. Identify
bottleneck
s
1. Define
success
A winning A/B test result is often just the beginning of the test’s value. There’s
much more to gain from a smart analysis of the results.
29. How the plan builds 5.
Optimise
4. Test
3.
Prioritise
2. Identify
bottleneck
s
1. Define
success
31. What to take away
• Exploration is a better test approach than refinement
• Align your tests to your business goals
• Identifying your barriers is vitally important
• Observe, hypothesise and form ideas
• Ideas can have many sources
• Write a plan and expect it to change
• Optimizely makes testing easier
• Dig deeper for learnings
• There are plenty of insights in failed tests
We have a diverse client base. We know that conversion in retail is fundamentally different to insurance, gaming etc. but we understand the nuances across industries (big or small) and can show the principles remain the same. The problems have been solved before.
Testing helps prove
What are we teaching you today:
1 how to build a plan through testing
We go in to lots of companies and they think they are doing testing but they are not – they are refining. When you refine, you take one variation and optimise it until it can’t get any better. Problem – is they have unbeatable landing pages but a terrible conversion rate.
The solution is exploration. Take a wider view and try out lots of ideas – let the data decide.
This is an important slide – pay attention, I will refer back to this again as I go along.
It’s funny, this may all sound obvious but it’s easily forgotten. By being more specific and defining this well, your testing becomes more structured.
Do your research before you start testing – analytics is a really good place to start, but you can also use info from surveys, focus groups and usability studies. You want a holistic view of what is going on on your website.
And next up we’ll go into real world examples of formulating a hypothesis.
Doing something like this is really easy within Optimizely, using the code and image from the Product Page and inserting it into the Test variation without needing dev team resource.
Now insurance is a high consideration product so this example isn’t as relevant to low price retail products. A complex product will generate more questions.
Another benefit of Optimizely is being able to test the mobile version of your website.
Testing and seeing one variation win is just the tip of the iceberg. You have to dig deeper to understand why.