Brett Jones - NEDMA May 2016 conference

1. Direct Mail Testing
& Key Metrics
Presented by: Brett Jones
May 17, 2016 2:30 – 3:20

2. Today We’ll Discuss
§ WHY its important to test
§ HOW to design a valid test
§ WHAT to test & how to measure results
1

3. WHY IT’S IMPORTANT TO TEST
2

4. 3 Ways to Make More Money
More Donors
• Acquisition Lists /
Select criteria for
Lapsed
• Package/Creative
• Lower initial ask
• Upfront / back-end
premium
More Gifts per
Donor
• RFM segmentation
to identify those
donors most likely
to give an
additional gift
• Follow up or
additional appeals
• Less aggressive
ask string
Higher Average Gift
• RFM segmentation
to identify those
with higher
previous gifts
• More aggressive
ask string
3
#4: Cutting Costs

5. The Testing Process
Determine
your
objective
Identify an
opportunity
Test your
hypothesis
Take action
based on
results
4
Possible Actions:
1. Retest
2. Rollout
3. Move on to
another
hypothesis

6. HOW TO DESIGN A
VALID TEST
5

7. HOW to design a valid test
What change do you want to
create?
Develop a hypothesis
Isolate a variable
& construct a
valid test
Execute Test
Determine
key metric
Read results
6

8. WHAT TO TEST
7

9. Focus on the things
that will make a difference
List, 60%Creative,
20%
Offer, 20%
8

10. Two Key Metrics
Response
Rate
Average
Gift
9
Costs

11. CREATIVE TESTS
10

13. Package Test in Acquisition:
Action v. Memo
QTY # Gifts %
Resp.
Avg.
Gift
Gross $ Net $ Net
Cost to
Acquire
“Action”
Control
29,905 384 1.28% $35.99 $13,819 ($4,723) ($12.29)
“Memo”
Test
29,901 208 0.70% $39.19 $8,152 ($9,999) ($48.07)
99% Confidence Level
What change did we want to create?
What was the hypothesis?
How would we determine success?
How did we construct a valid test?
What won? What might we do next?

14. Picture
Window
vs.
Standard
OE

15. Which format won?
Segment QTY # Gifts % Resp. Avg.
Gift
Gross $ Gross
Rev/M
Control
OE
13,033 229 2.29% $48.09 $14,380 $1,103
Picture
Window
13,033 373 2.86% $44.36 $16,547 $1,270
99% Confidence
Level
What change did we want to create?
Our hypothesis. What variable did we isolate?
How would we determine success?
What we will do next?
Why / When
to use this
metric

17. Component Test in Acquisition:
RE v. BRE
Segment QTY # Gifts % Resp. Avg. Gift Gross $
BRE 9,931 97 0.98% $36.81 $3,571
RE 9,931 69 0.69% $41.09 $2,835
95% Confidence Level
What change did we want to create?
What was the hypothesis?
What won?
What we will do next?

18. Format Test:
Attached vs.
Separate Reply

19. Format Test:
Attached vs. Separate Reply
Segment QTY # Gifts % Resp. Avg. Gift Gross $ Cost/Piece
Separate
Reply
63,239 1,034 1.60% $66.87 $69,141 $0.36
Attached
Reply
63,773 1,176 1.84% $75.64 $88,962 $0.30
99% Confidence Level
What change did we want to create?
What was the hypothesis?
What about the increase in the average gift?
What won?
What we will do next?

20. OFFER TESTS
19

21. Longwood Gardens
Can an offer be too rich?
20
Test Offer:
2 Guest Passes
Control Offer:
2 Additional Guest Passes

22. Segment QTY # Gifts % Resp. Avg.
Gift
Gross $ Gross
Rev/M
Control:
MORE
passes
56,763 484 0.85% $75.63 $36,605 $645
Test :
fewer
passes
56,761 558 0.98% $74.98 $41,840 $737
What change did we want to create?
What was the hypothesis?
What would we measure to determine success?
What variable did we isolate?
What won? Why?
95% Confidence Level
Longwood Gardens
Offer Test

23. Any Amount Makes You a Member
22
Sometimes you need to evaluate
results over a l-o-n-g period of time

24. Open Ask Test Results
§ Response Rate increased by 122%
§ Average Gift fell from $55 to $28
§ Net Cost to Acquire a member cut in half
§ Minimal reduction in retention rate Year 2
23

25. LIST TESTS
24

26. Segment Test
Segment QTY #
Gifts
%
Resp.
Avg.
Gift
Gross $ Net Rev.
Recently
Lapsed
9,682 139 1.44% $48.10 $6,686 $1,861
Longer
Lapsed
21,681 197 0.91% $52.56 $10,354 ($451)
Would you
mail this
group
again?
How would you decide?

27. Segment Test
Segment QTY #
Gifts
%
Resp.
Avg.
Gift
Gross $ Net Rev. Net
Cost to
Acquire
Acquisition 63,521 381 0.60% $46.49 $17,713 ($13,984) ($36.70)
Recently
Lapsed
9,682 139 1.44% $48.10 $6,686 $1,861 +
Longer
Lapsed
21,681 197 0.91% $52.56 $10,354 ($451) ($2.29)
Would you
mail this
group
again?
Goal: More New Donors
Key Metric: Net Cost to Acquire Longer Lapsed donor vs. Acquisition

28. Every acquisition mailing is an
opportunity to tests new lists
Performance Risk

29. A Balanced List Plan
Test Lists
10% max
Retests
Approx. 10%
Core Lists
80% of names

30. § List testing opportunity #1:
Identify lists that are mediocre performers whose response may be
increased with a tighter select
List Segment Historical Resp. % Avg. Gift
Habitat For Humanity/24 Mos. .87% $48.09
List Segment % Resp. Avg. Gift
Habitat For Humanity/12 Mos. 1.01% $48.07
Current Performance
Sample Test Results
If the available universe doesn’t shrink too much, you may be
better served by tightening the select criteria

31. § List testing opportunity #2: The other side of the coin
Identify lists that are very strong performers that could generate
additional members with an expanded select (larger universe) and
comparable (or acceptable) response
List Segment Mailing
Universe
Historical
Resp. %
Members
World Wildlife Fund/12 Mos. 4,000 1.03% 41
World Wildlife Fund/12
Mos.
Mailing
Universe
% Resp. Members
World Wildlife Fund/24
Mos.
7,000 .91% 63
Current Performance
Sample Test Results
Sometimes the greater good is served by a lower response
rate on a larger universe of prospects – ultimately delivering
you more new donors.

32. Is an UNsuccessful test
one that doesn’t give you the
answer you expected?

33. What if a test loses?
What if I’m “too small” to test?

34. 33
Remember…

35. Brett Jones
Vice President, Fundraising
DMW Fundraising
bjones@dmwfundraising.com
Thank you!