2. What is testing?
Testing is a marketing experiment in a live environment:
• Observing what customers actually do rather than
what they say they are going to do
• Monitoring how results vary as various components of
a campaign are altered.
• Using the results of testing to:
– predict the outcome of future activity.
– Improve our marketing performance
Australian Direct Marketing Association
3. 3 approaches
1. Test on a small scale
Roll-out first then roll-out
Test
2. Do testing on a portion
Live campaign of a live campaign
Test
3. Same approach to all
Live campaign
? then analyse results among
different segments to spot
variations
Australian Direct Marketing Association
4. Summary of testing options
Test the important things…..
Message components Delivery components
• Product • Targeting & segmentation
• Proposition • Communication channels
• Offer • Format
• Creative • Timing
• Call to action
Australian Direct Marketing Association
5. Testing case study
• Two simple objectives
– Improve response rates
– Increase amount donated
• Understanding donor segments
– Relationship to disease
– Value
With permission from Salmat
Australian Direct Marketing Association
6. Test different propositions for each segment
• Relationship to disease
– Have the disease
– Parent of someone with the disease
– Relative / friend of someone with the disease
– No relationship to the disease
With permission from Salmat
Australian Direct Marketing Association
7. Personalisation case study
• Value
– Variable donations boxes based on last donation, increased in
increments of 20%
With permission from Salmat
Australian Direct Marketing Association
9. Example: Landing Page Test Options
Page components
1. Headline
2. Secondary Headline
3 Body Copy
4 Social Proof
5/6 Call to Action
7 Navigation
8 Image
9 Above/below the Fold
Australian Direct Marketing Association
10. Guidelines for accurate testing
• Ensure you always have a “control”
– So you have a benchmark for comparison
• Have a big enough sample to test with
– To produce reliable results
• Ensure your test sample is not skewed at all (compared to
control)
– Use a randomly-selected sample
• Decide how you are going to measure performance:
–The measurement chosen must prove/disprove your hypothesis
Australian Direct Marketing Association
11. The importance of a control
Offer test Here there is no control:
Response
rate
- A separate offer has been
run in each month
New offer A
Standard offer New offer B
- Offer A appears to have out-
performed the current offer
- Offer B appears to have
performed worse
= Offer A appears to win
May June July
Australian Direct Marketing Association
12. The importance of a control
Offer test Introduction of control:
Response
rate
- The current offer has been
run in each month as a
New offer A benchmark
Standard offer New offer B
- Offer A did not perform as
well as the current offer
- Offer B performed better than
the current offer
May June July
= Offer B is the real winner
Australian Direct Marketing Association
13. The “significance” of a test result
Response
rate The test out-performed the control by 0.3%
1.7%
1.4%
Should you adopt New Offer A – or could it
be just a chance result?
We need to know whether these results are
likely to be repeated if we ran the test again
Control New Offer A
Australian Direct Marketing Association
14. Having a Big Enough Sample
• In direct marketing, your sample is
Required Sample Quantity
typically a subset of your customer
base or contact list
• In digital, your sample is typically the
volume of unique visitors or unique
impressions accumulated over a
period of time
• You need a quantity of sample that
reflects your desire to detect even 50%$ 60%$ 70%$ 80%$ 90%$ 95%$ 99%$
small differences between your test
and control groups Ability to Detect Statistically
Significant Differences
Australian Direct Marketing Association
15. Not Just Statistical Significance
Do a sense-check when interpreting results:
• What was the competition doing when this test was running?
• Just because this worked in one location does it mean it will work in
another?
• The offer was successful in Summer – would it still work in Winter?
• Were there any other abnormal factors in the marketplace which
might have affected the response?
Australian Direct Marketing Association
16. Not Just Statistical Significance
Did we make money?
When we balance the costs of a campaign against the
incremental volume of sales – was it worth it?
What should we test next?
e.g. offer A performed better than offer B, what will beat
offer A?
Australian Direct Marketing Association
18. Tracking the customer journey
A buyer typically
goes through several
stages before they buy
Awareness Interest Desire Action
…which means we should have
measures to track
performance for each stage
Reach Influence Closure
Australian Direct Marketing Association
19. Example: eCommerce
Awareness Daily
Unique
Visitors
Interest Product
Page
Views
Desire Completes
a
Registra:on
Form
Action Purchase
Australian Direct Marketing Association
20. Tracking issues
How do you match response to source when:
– Your campaign runs in several channels
simultaneously
– Your product is sold through a third party sales channel
– Your product has a long consideration period
Australian Direct Marketing Association
21. Common tracking options
Outbound Channel Response Channel
Offline Unique campaign and
Call-Centre comms channel
Broadcast
specific phone numbers
Mail
Unique redemption
In-Store codes
Display
Media
Specific Search Call to
Website Action
Australian Direct Marketing Association
22. Indirect sales – a missing link
Browse Known
Clicks/visits
Buy Unknown
Sales
Enquire Known
Calls
Australian Direct Marketing Association
23. Long sales cycles
Sales
Time
What does this response curve look like for your product?
• Track enquiries & sales over a long period
• Establish a pattern
• Create a rule of thumb
Australian Direct Marketing Association
24. Rule of Thumb Approach
• Can be used for indirect sales as well as an ‘early read’ for long
campaign cycles
• Typical approach:
1. Establish a ratio for website visits or calls to reseller enquiries/
sales
2. Establish a pre-campaign baseline for calls and website visits
3. Measure the uplift in calls/visits during and following the
promotion
4. Extrapolate to sales using typical ratio
Australian Direct Marketing Association