3. Today’s team
Dr. Flint McGlaughlin Jon Powell
Managing Director Senior Manager
Research and Strategy
#webclinic
4. The Challenge
Q: Are there metrics your organization does NOT monitor, only because they are
not set up properly?
4
#webclinic
5. Background and Test Design
Experiment ID: REGOnline Homepage Test
Location: MarketingExperiments Research Library
Test Protocol Number: TP1427
Research Notes:
Background: REGOnline is event management software that lets users create
online registration forms and event websites to manage their events.
Goal: To increase number of completed leads on homepage.
Primary research question: Which page will generate the greatest number of
leads?
Approach: A/B multifactor split test
#webclinic
6. Experiment: Control
Control - Homepage
• Our researchers
hypothesized that we could
increase the appeal
associated with the value
proposition of this offer by
focusing more on the
product and its specific
features and benefits.
#webclinic
7. Experiment: Treatment
Treatment - Homepage
• Headline was written to focus
more on the product.
• Specific features and benefits
are utilized to express the
value.
• The page emphasizes “Free
Access.”
• Also, ensured that this value
was being communicated in
subsequent steps.
7
#webclinic
9. Experiment: Results
24.5% Decrease in Conversion
The Treatment generated 24.5% less completed leads
Conversion
Versions Rel. diff
Rate
Control – Two-step homepage 2.3% -
Treatment – Three-step homepage 1.7% -24.5%
What youthe amount of form fieldsspite of first step,clearer valuestill
reducing
need to understand: In
in the
having a
the control
and
outperformed the treatment.
#webclinic
10. Experiment #2: Background
Homepage from
Previous Test
• Before we could get a lift, we
needed to learn more about
the prospects coming to this
site.
24.5%
Decrease in Conversion
• We decided to use one of
their SEO pages as a research
window into the cognitive
psychology of the customer’s
motivation.
10
#webclinic
11. Experiment #2: Background
Experiment ID: REGonline SEO landing page test
Location: MarketingExperiments Research Library
Test Protocol Number: TP3055
Research Notes:
Background: A technology and media company specializing in online registration and
event management software.
Goal: To increase the amount of leads generated online.
Primary research question: Which online capture process will generate the higher
addressable lead rate?
Approach: A/B multifactor split test
11
#webclinic
12. Experiment #2: Control
SEO Landing Page
• This landing page was
offering the same
product as the home
page but dealt with a
smaller subset of
visitors who matched
the profile of those
coming to the
homepage.
• Our researchers could
test here without the
negative consequences
of hurting conversion
on the homepage.
12
#webclinic
13. Experiment #2: Treatment
Treatment SEO Landing Page
• For our first test on this
page, we tested
focusing on how this
product made the
process of creating
registration forms
easier and could cut
the prospects’ time in
half…
• …and yet it still had a
robust functionality.
#webclinic
14. Experiment #2: Side-by-side
Control Treatment
Which copy language will generate the most leads?
14
#webclinic
15. Experiment #2: Results
548% Increase in Complete Leads
The new page’s conversion rate increased by 548.46%
Conversion Rate Relative Statistical Level
Design (%) Difference of Confidence
Original Page 0.7% - -
Treatment 4.8% 548% 99%
What you need to understand: Bythe treatment wasthis product made
creating registration forms easier,
focusing on how
able to increase
step-level clickthrough rate by 1,312%, and completed leads captured
by 548%.
15
#webclinic
16. Experiment #2: Final Results
Original Homepage New Homepage Test
SEO Page Test
548% Learning Learning 90%
• We were able to take what we learned about the motivations of their
customers from testing on the SEO landing page and apply it to the
homepage, which generated a 90% increase in leads captured.
16
#webclinic
17. What we discovered
F Key Principles
1. The goal of all customer research is to enable the marketer to
predict customer behavior.
2. Therefore, the primary usefulness of metrics is not in answering
“how many?” but rather in answering, “why so?”
3. Ultimately, metrics enable the marketer to see the cognitive trail
left by the visitor’s mind.
#webclinic
18. How do we cut through it all?
• The problem is not typically getting sufficient data from you metrics
software. Rather, the challenge is making sense of it.
18
#webclinic
20. Today, we will walk through a simple 5-step process for
translating raw testing data into predictive power
20
#webclinic
21. Translating Raw Data to Predictive Power
F Key Steps
1. Establish Visibility – Ensure that your metric platforms are able to track the
four primary types of analytics:
21
#webclinic
22. STEP 1: Establish Visibility
Types of Analytics – Visual
Page views referrers
search terms
visitor sessions languages
Amount Source
returning visitors organizations
impressions geographic location
Entry pages
Sign-ups Orders
exit pages
browsers Number of page views
Nature Results
Screen resolution
time on page Click trails
Load errors Most requested pages
22
#webclinic
23. Translating Raw Data to Predictive Power
F Key Steps
1. Establish Visibility – Ensure that your metric platforms are able to track the
four primary types of analytics:
Amount – How many instances of a particular action are occurring?
Source – Where are prospects coming from?
Nature – What are prospects experiencing on your site?
Results – What are prospects doing on your site?
23
#webclinic
24. Translating Raw Data to Predictive Power
F Key Steps
1. Establish Visibility – Ensure that your metric platforms are able to track the
four primary types of analytics: (1) Amount, (2) Source, (3) Nature, (4) Results.
2. Determine Objective – Determine the exact research question you are setting
out to answer with your metrics.
24
#webclinic
25. STEP 2: Determine the Objective
The Research Question
1. Whether you are running a live test or conducting a forensics metrics analysis,
your research and metrics analysis must be grounded in a properly framed
Research Question.
2. A properly framed Research Question is a question of “which” and sets out to
identify an alternative (treatment) that performs better than the control.
Example:
Not this..
What is the best price for product X?
But this…
Which of these three price points is best for product X?
* Depending on the data available, forensics data is often grounded in a research question of “what?” rather than “which”.
25
#webclinic
26. STEP 2: Determine the Objective
Audience Exercise
? How would you refine the following three research questions?
1. What is the best headline for my landing page?
2. Why do I have such a high bounce rate on my offer page?
3. How many objectives should I have on my homepage?
26
#webclinic
27. STEP 2: Determine the Objective
The Research Question
1. Often, metrics can also Unique visits
be utilized to Flights
40,607,893
determine the most Hotels
32%
effective research 14,185,646
Autos Not all visitors go
questions you should 7,729,403 through each of
these steps
be asking. Activities
9,167,901
60%
Travelers
2. Metrics can be a 73%
12,883,177
Summary
window into key gaps 58%
7,717,122
into your customer Login
5,665,020
76%
theory and ultimately Contact
3,260,292
into the highest 71%
Payment
potential revenue 2,484,236
opportunities for Completion Rate
once process begins
4% 1,766,609
marketing efforts.
27
#webclinic
28. STEP 2: Determine the Objective
Example Case Study – Experiment Background
Experiment ID: (Protected)
Location: MarketingExperiments Research Library
Test Protocol Number: TP1305
Research Notes:
Background: A website that sells retail and wholesale collector items
Goal: To increase conversion rate
Primary research question: Which version of second step in the conversion
funnel will produce the highest conversion rate?
Approach: A/B variable cluster split test that focused on reducing anxiety
through credibility indicators, copy, and re-organization of existing page
elements
28
#webclinic
29. STEP 2: Determine the Objective
Example Case Study – Experiment Background
Fallout Report: New Customers • When we analyzed the metrics, we
realized there were leaks throughout
the checkout process, the credit card
submission page stood out as low
cost opportunity for immediate
return.
• When we analyzed the metrics even
further, we saw that this step also
had the highest lost revenue per cart
(more than double of any other step).
• From this, we hypothesized that
optimizing this step would have the
highest potential return on our
efforts.
29
#webclinic
30. STEP 2: Determine the Objective
Example Case Study – Experiment Control
Control What might be causing the
fallout?
• It is unclear why the credit
card is required when
payment method is
different.
• The complexity of the
Purchase Agreement Terms’
causes confusion and
concern.
• There is no indication that
my credit card information
is secure.
30
#webclinic
31. STEP 2: Determine the Objective
Example Case Study – Experiment Treatment
Treatment How we addressed the issues:
• Third-party security indicators
have been added.
• Clearer explanation of why a
credit card is required and that
it will not be charged.
• “Satisfaction Guaranteed”
promise is emphasized.
31
#webclinic
32. STEP 2: Determine the Objective
Example Case Study – Experiment Results
5% Increase in total conversion
The new credit card page increased conversion by 4.51%
Design Conversion Rate
Control 82.33%
Treatment 86.04%
Relative Difference 4.51%
What youthis specific step in theWhile it mighttoseem resulted in a projected
choosing
need to understand:
sales funnel test
like a small increase,
$500,000+ increase in revenue per year. This underscores the potential
impact of a properly identified research question.
32
#webclinic
33. Translating Raw Data to Predictive Power
F Key Steps
1. Establish Visibility – Ensure that your metric platforms are able to track the
four primary types of analytics: (1) Amount, (2) Source, (3) Nature, (4) Results.
2. Determine Objective – Determine the exact research question you are setting
out to answer with your metrics.
3. Track and Measure – Track and measure the appropriate metrics that will
provide you with the answer to your determined research question.
33
#webclinic
34. STEP 3: Track and Measure
Primary and Secondary Metrics
1. Primary “Test” Metrics: The
essential metrics that enable
you to answer the research
question
Primary
Metrics 2. Secondary Metrics: The
additional metrics you can
utilize to help interpret the
Secondary results of your primary metrics
Metrics
34
#webclinic
35. STEP 3: Track and Measure
Primary Metrics – Examples
Example #1:
Research Question: Which headline will generate the most subscriptions?
Primary Metrics: Visits, subscriptions subscription rate (%)
Example #2:
Research Question: Which PPC ad will generate the most qualified traffic?
Primary Metrics: Ad spend, conversions cost per acquisition ($)
Example #3:
Research Question: Which page will generate the most Facebook fans?
Primary Metrics: Visitors, clicks on the “Like” button fans per visitor (%)
35
#webclinic
36. STEP 3: Track and Measure
Secondary Metrics – Examples
Secondary Metric Potential Insights
Are visitors engaged with the content?
Time on page
Are they confused with the process?
What are visitors interested in?
Click tracking Are they confused with the process?
Is there a lack of relevance to visitors?
Bounce rate Are there too many distractions? Is there too
much (or little) information?
What motivates individual visitor types?
Segment-level data Where are the deeper optimization opportunities?
Form event tracking What form fields cause anxiety or confusion?
How much friction will your visitor put up with?
Traffic patterns Who is coming and where are they coming from?
Can we be more relevant to the visitor?
36
#webclinic
37. STEP 3: Track and Measure
Example Case Study – Experiment Background
Experiment ID: (Protected)
Location: MarketingExperiments Research Library
Test Protocol Number: TP1341
Research Notes:
Background: A company offering dedicated hosting services
Goal: To increase the number of leads
Primary research question: Which page design will generate the greater
number of leads?
Approach: A/B multi-factor split test (radical redesign)
37
#webclinic
38. STEP 3: Track and Measure
Example Case Study – Experiment Treatments
Control Treatment
Let’s consider both the primary and secondary metrics
utilized for this test…
38
#webclinic
39. STEP 3: Track and Measure
Example Case Study – Experiment Metrics
Control Treatment
Research Question: Which page design will generate the
greater number of leads?
Primary Metrics Primary Metrics
Visits = 31,400* Visits = 30,560*
leads = 628* Leads = 1,764*
CR = 2.0% CR = 5.7%
Answer: The treatment design will generate 188% more
leads.
* Numbers have been anonymized 39
#webclinic
40. STEP 3: Track and Measure
Example Case Study – Experiment Metrics
• In addition to tracking the
primary metrics, the research
analysts installed some
secondary event tracking
metrics.
• On this page, there were six
expandable sections of copy
featuring different elements of
the product value proposition.
• By monitoring the specific clicks
of visitors on this page, we were
better able to understand what
aspect of this product’s value
proposition was most appealing
to the visitor.
40
#webclinic
41. Translating Raw Data to Predictive Power
F Key Steps
1. Establish Visibility – Ensure that your metric platforms are able to track the
four primary types of analytics: (1) Amount, (2) Source, (3) Nature, (4) Results.
2. Determine Objective – Determine the exact research question you are setting
out to answer with your metrics.
3. Track and Measure – Track and measure the appropriate metrics that will
provide you with the answer to your determined research question.
4. Monitor Anomalies – Monitor the data for any anomalies that might indicate a
validity threat.
41
#webclinic
42. STEP 4: Monitor Anomalies
Audience Question
? What wrong with this test data set?
19.00%
17.00%
15.00%
Conversion Rate
13.00%
11.00% Control
9.00% Treatment 3
7.00%
5.00%
3.00%
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11
Test Duration
42
#webclinic
43. #webclinic
campaign
rates, etc.)
Validity Threats
to a specific online
rates, sales, average
• A more subtle clue is a
of response visitors are
temporary spikes in the
having to a specific online
amount of traffic or views
campaign (e.g., conversion
• Monitor for unexplainable
purchase amounts, bounce
noticeable shift in the kind
3.000
0.000
1.000
2.000
4.000
(3.000)
(2.000)
(1.000)
Saturday, October 11, 2008
Sunday, October 12, 2008
Monday, October 13, 2008
Tuesday, October 14, 2008
Wednesday, October 15, 2008
Thursday, October 16, 2008
Friday, October 17, 2008
STEP 4: Monitor Anomalies
YES
Saturday, October 18, 2008
Sunday, October 19, 2008
Monday, October 20, 2008
Tuesday, October 21, 2008
Wednesday, October 22, 2008
Thursday, October 23, 2008
Friday, October 24, 2008
Saturday, October 25, 2008
Sunday, October 26, 2008
Monday, October 27, 2008
Tuesday, October 28, 2008
NO
Wednesday, October 29, 2008
Thursday, October 30, 2008
Friday, October 31, 2008
Saturday, November 01, 2008
Sunday, November 02, 2008
Monday, November 03, 2008
Tuesday, November 04, 2008
Wednesday, November 05, 2008
Standardized Conversion Rate
Thursday, November 06, 2008
NO
Friday, November 07, 2008
Saturday, November 08, 2008
Sunday, November 09, 2008
Monday, November 10, 2008
Tuesday, November 11, 2008
Wednesday, November 12, 2008
Thursday, November 13, 2008
Graphed results of a 4-week email test with an ecommerce retailer:
Normalized
Normalized B
Normalized Traffic
Normalized Traffic B
43
44. STEP 4: Monitor Anomalies
Validity Threats
Anomalies in your metrics can indicate that there may be validity threats in your
tests and data. Be sure to check for the following validity threats should you
encounter any anomaly.
History Effect – when a test variable is affected by an extraneous variable
associated with the passage of time
Instrumentation Effect – when a test variable is affected by a change in the
measurement instrument
Selection Effect – when a test variable is affected by different types of
subjects not being properly distributed among
experimental treatments
For more on validity threats, see our previous Web clinic replay:
“Bad Data: The 3 validity threats that make your tests look conclusive (when they are deeply flawed).”
44
#webclinic
45. Translating Raw Data to Predictive Power
F Key Steps
1. Establish Visibility – Ensure that your metric platforms are able to track the
four primary types of analytics: (1) Amount, (2) Source, (3) Nature, (4) Results.
2. Determine Objective – Determine the exact research question you are setting
out to answer with your metrics.
3. Track and Measure – Track and measure the appropriate metrics that will
provide you with the answer to your determined research question.
4. Monitor Anomalies – Monitor the data for any anomalies that might indicate a
validity threat.
5. Interpret Data – Interpret the data by moving from “Which?” to “Why?” to
“What?” to “Where?”.
45
#webclinic
46. STEP 5: Interpret Data
From Customer Behavior to Customer Theory
Which? Why? What?
Customer Behavior Customer Theory
Which headline will Why this headline?
What does my customer
generate a higher want the most?
response?
Why this testimonial?
Which testimonial will What makes my customer
generate the most especially anxious?
response?
Which call to action will Why this call-to-action? What is my customer’s position in
generate a higher the sequence of micro-yeses?
response?
46
#webclinic
47. STEP 5: Interpret Data
Example Case Study
Again, test results are
interpreted and the next
round of testing is started
for this page
201% 2% 29%
Test results are interpreted and Test is again interpreted and
second test was created based on transferrable principles are
the analyst’s observations applied to other offer pages
47
#webclinic
48. STEP 5: Interpret Data
Where else can we apply this data?
• The discoveries and insights about 451%
customer motivation from the
three prior tests were applied to
other landing pages and used to
optimize PPC campaigns.
• The purposeful effort to identify 302%
and selectively apply these
transferrable insights led to
widespread optimization gains .
257% 28% 603%
48
#webclinic
49. Baltimore Training Week
Save $100 off any workshop
Promo Code: 284-WS-2022
July 30 - August 1
www.meclabs.com/BTW
49
#webclinic
50. Summary: Putting it all together
F Key Principles
1. The goal of all customer research is to enable the marketer to predict
customer behavior.
2. Therefore, the primary usefulness of metrics is not in answering “how
many?” but rather in answering, “why so?”
3. Ultimately, metrics enable the marketer to see the cognitive trail left by the
visitor’s mind.
50
#webclinic
51. Summary: Putting it all together
F Key Steps
1. Establish Visibility – Ensure that your metric platforms are able to track the four
primary types of analytics: (1) Amount, (2) Source, (3) Nature, (4) Results.
2. Determine Objective – Determine the exact research question you are setting out to
answer with your metrics.
3. Track and Measure – Track and measure the appropriate metrics that will provide
you with the answer to your determined research question.
4. Monitor Anomalies – Monitor the data for any anomalies that might indicate a
validity threat.
5. Interpret Data – Interpret the data by moving from “Which?” to “Why?” to “What?”
to “Where?”.
51
#webclinic
52. Audience Question
How can I track and integrate social media metrics
? into my web analytics?
-Anne
52
#webclinic
53. Audience Question
Is Google Analytics "good enough“ to measure
? everything I need?
-Lou
53
#webclinic
54. Audience Question
What is the best method for calculating
? incremental click costs for low volume keywords?
- Don
54
#webclinic
55. Audience
How should I interpret bounce rates?
?
- Steve
55
#webclinic