How confident are you of understanding how (and why) customers react to your campaigns?
In his ClickZ presentation, Taylor Kennedy, Senior Manager of Training, shared lessons that will optimize ecommerce campaigns to help you identify principles that improve your own testing strategy, while getting a better understanding of your customer.
ClickZ Live -- Chicago 2014: 3 Common Testing Mistakes (and how to avoid them)
1. November 3–6, 2014
#CZLCHI | @ClickZLive
The Global Conference Series Designed by Digital Marketers, for Digital Marketers
3 Common Mistakes in Online Testing
(And How to Avoid Them)
Taylor Kennedy
MECLABS
Senior Manager, Optimization & Strategy
2. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
30%
16%
Do you understand your results?
60%
12%
43%
8%
62%
6%
5. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
You’re (probably) testing wrong if you:
•
Are stuck optimizing a part of your funnel that lacks significant impact on the bottom line
•
Have been blindly following “best practices”
•
Are not further establishing customer theory with each test
14. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Cart Test 2: Results
Results were flat, even trending downward.
Shopping Cart
Conversion Rate
Rel. Difference
Level of Confidence
Control
13.7%
NA
---
Treatment
11.1%
-19.26%
88%
?
15. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Why aren’t tests getting results?
5.7%
41%
All visitors
Add to cart
Payment info
Convert
40%
16. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Through proper planning you:
Identifythe highest impact area of your site to test.
Findpotential validity threats linked to normal traffic that may impact results.
Discover visitor behavior that helps you create long-term test plans.
19. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Product Page Test 1: Results
•
“Add to Cart” increased by over 15%
•
Conversion increased by 40%
Shopping Cart
Conversion Rate
Rel. Difference
Level of Confidence
Control
1.07%
NA
---
Treatment
1.50%
40.05%
98%
!
20. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
What motivated the cart test?
A.
Needed improvements were obvious
B.
Our client or boss told us to
C.
Testing close to the conversion event has the biggest ability to increase revenue
D.
The data told us to
21. November 3–6, 2014 | #CZLCHI | @ClickZLive
You’ve been following “best practices”
Case study examples
22. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
How are successful and unsuccessful ecommerce companies approaching testing?
MarketingSherpa asked marketers:
29. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
How often do you hear the following?
“Keep your call-to-action above the fold”
“Video converts”
“Use a compelling image”
“Mobile-friendly is critical”
“Include privacy and security statements to establish trust”
37. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Mobile Redesign Test: Results
Results were significantly negative.
Shopping Cart
Conversion Rate
Rel. Difference
Level of Confidence
Control
0.5%
NA
---
Treatment
0.3%
-40.75%
96%
!
38. November 3–6, 2014 | #CZLCHI | @ClickZLive
Redefine best practice:
•
There are no shortcuts to customer wisdom.
•
Just because a test worked for someone else, doesn’t mean it will work for you.
•
Rigorous, methodology-driven, hypothesis testing ensures you are generating real customer insights.
45. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A bad hypothesis:
“By reducing friction on the page associated with product selection, we will see an increase in clickthrough.”
•
Unclear link to a particular problem
•
Not related to specific elements on a particular page or process
•
Not linked to information or data about the page
46. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A bad hypothesis:
“By reducing friction on the page associated with product selection, we will see an increase in clickthrough.”
•
Unclear link to a particular problem
•
Not related to specific elements on a particular page or process
•
Not linked to information or data about the page
47. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A bad hypothesis:
“By reducing friction on the page associated with product selection, we will see an increase in clickthrough.”
•
Unclear link to a particular problem
•
Not related to specific elements on a particular page or process
•
Not linked to information or data about the page
48. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A bad hypothesis:
“By reducing friction on the page associated with product selection, we will see an increasein clickthrough.”
•
Unclear link to a particular problem
•
Not related to specific elements on a particular page or process
•
Not linked to information or data about the page
49. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A good hypothesis:
CHANGING __________________ INTO__________________
WILL_______________________
What your analysis indicates is the problem
What change you think will solve the problem
The effect it will have on your KPI
50. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A good hypothesis:
“Changing the presentation of products from a single column format into a dual column format will increase the number of product options visible and boost clickthrough to the product details page”
51. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A good hypothesis:
“Changing the presentation of products from a single column format into a dual column format willincrease the number of product options visible and boost clickthrough to the product details page”
52. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A good hypothesis:
“Changing the presentation of products from a single column format into a dual column format will increase the number of product options visible and boost clickthrough to the product details page.”
53. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
A good hypothesis is:
•
A proposed statement
•
Made on the basis of limited evidence related to a specific problem
•
Can be proved or disproved
•
Used as a starting point for further investigation
56. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Motivation Test: Control
Hypothesis:
Changing the right-column form intoa call-to- action requiring no personal information will reduce friction and boost clickthrough to the subsequent page.
61. @OptimizeDude
November 3–6, 2014 | #CZLCHI | @ClickZLive
Remember:
•
Proper planning allows you to run tests with clear outcomes predetermined in advance
•
Formulate test sequences that generate big-picture customer analysis
!
62. November 3–6, 2014 | #CZLCHI | @ClickZLive
Key takeaways
Follow a “testing process.” Don’t just “test.”
1.
Always perform at least a basic site data analysis (funnel analysis) before beginning testing.
2.
Analyze each page through the mind of your prospect to identify areas of opportunity for testing.
3.
Formulate hypotheses that enable you to learn from each and every test. That will lead you to a more complete picture of your ideal prospect.
63. November 3–6, 2014 | #CZLCHI | @ClickZLive
http://bit.ly/OptimizeTK
Presentation Link
Hinweis der Redaktion
“Preparing online tests the right way” – alternate name for the presentation
I like “Preparing…” better as a title – KC
Have you ever run a test where you don’t understand the results?
The biggest enemy to your online testing efforts is the inability to explain your own results
How often as an online tester are you confused about why a test showed a particular result?
Without being able to properly explain your test results, you will never get the buy-in you need to unlock the true power of testing
Have you ever run a test where you don’t understand the results?
The biggest enemy to your online testing efforts is the inability to explain your own results
How often as an online tester are you confused about why a test showed a particular result?
Without being able to properly explain your test results, you will never get the buy-in you need to unlock the true power of testing
If you’ve had a result you didn’t understand more than a handful of times, then you’re testing wrong
While not a big Jeff Foxworthy fan myself, I can’t help think of this slide as a “you might be a redneck if … sort of slide” because I’m going to outline some common mistakes that all marketers have likely made in their experiences running online tests.
I’ve personally made these mistakes, and I want to ensure that I teach you something that will help you to avoid them.
While this may seem like common sense, you would be amazed by how many partners that I have worked with that do not follow a repeatable process
By focusing on proper discovery and planning, you can ensure that you will understand your test results the vast majority of the time by properly accounting for variables under your control.
While this may seem like common sense, you would be amazed by how many partners that I have worked with that do not follow a repeatable process
By focusing on proper discovery and planning you can ensure that you will understand your test results the vast majority of the time by properly accounting for variables under your control.
Total price is extremely unclear, order summary is at the bottom of the page
Added clarity
While this may seem like common sense, you would be amazed by how many partners that I have worked with that do not follow a repeatable process
By focusing on proper discovery and planning you can ensure that you will understand your test results the vast majority of the time by properly accounting for variables under your control.
We only got to do a few of the things we wanted in our first test, so now were going to do a variable cluster test and change multiple elements to add clarity
Brought the cart summary above the fold
Clarified and streamlined the shopping cart experience
Added dynamic information about shipping options
Flat again
It became obvious when we finally got our data analysis back…motivation to convert was so high in the cart, that the changes we made did not produce any significant impact.
So we had to move up in the funnel…
Link to blog post or other resource with more information
Once we saw the picture that the data painted, we knew where our efforts would have the most impact. We moved up in the funnel to the product page.
Long copy, with large blocks of information below the fold
Prioritized “customer service” and “product quality” content that we hypothesized would affect the prospects buying decision
Flat again
We didn’t answer D, but whatever situation you are in, you should be looking to answer D
Displayed limited options for “default” smartphones
Displayed limited options for “default” smartphones
Displayed limited options for “default” smartphones
Displayed limited options for “default” smartphones
Displayed limited options for “default” smartphones
Displayed limited options for “default” smartphones
Could put common statements on this slide or show a template broken out
Expanded smartphone options to include a longer page with additional popular products
Expanded smartphone options to include a longer page with additional popular products
Expanded smartphone options to include a longer page with additional popular products
Expanded smartphone options to include a longer page with additional popular products
Flat again
Displays numerous product options and categories to select from
Displays numerous product options and categories to select from
Flat again
We didn’t answer D, but whatever situation you are in, you should be looking to answer D
We didn’t answer D, but whatever situation you are in, you should be looking to answer D
We didn’t answer D, but whatever situation you are in, you should be looking to answer D
We didn’t answer D, but whatever situation you are in, you should be looking to answer D
We didn’t answer D, but whatever situation you are in, you should be looking to answer D
We didn’t answer D, but whatever situation you are in, you should be looking to answer D
We didn’t answer D, but whatever situation you are in, you should be looking to answer D
Single product ecommerce flow that required visitors to input their delivery information before full price disclosure
Observed a high bounce rate on the landing page, with relatively high completion rates for the rest of the funnel
What is the number 1 thing visitors in the ecommerce space are traditionally looking for? – Price!
Single product ecommerce flow that required visitors to input their delivery information before full price disclosure
Observed a high bounce rate on the landing page, with relatively high completion rates for the rest of the funnel
What is the number 1 thing visitors in the ecommerce space are traditionally looking for? – Price!
If we open the gates, we’ll drive a ton more people to the product details page
The results were exactly what was expected, and justified the customer theory that visitors weren’t actually interested in the product, but were interested instead in the pricing.