1. Test, Rinse, Repeat
The ins and outs of digital testing
FRIDAY, OCTOBER 11, 2019
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
2. Meet & Greet
Lindsey Twombly
DIGITAL PRODUCTION SPECIALIST
LAUTMAN MASKA NEILL & COMPANY
Ann Crowley
VICE PRESIDENT, MEMBERSHIP & ONLINE STRATEGY
HUMAN RIGHTS CAMPAIGN
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
3. Today’s Agenda
1. Why testing? (<5 minutes)
2. Getting your boss on board with testing (10 minutes)
3. Real-life examples & how-to walk-thrus (30 minutes)
4. Open discussion & idea exchange (15 minutes)
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
4. I. Why Should You Test?
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
5. Why Should You Test?
Goals of testing:
● Keep up with the ever-changing digital landscape
(what works today may not work tomorrow)
● Improve Your Digital Program
○ More email opens & clicks (which help
deliverability)
○ Greater page completion rates
○ Increased actions and donations
○ Better engaged list &
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
6. Terminology
A/B Test vs. 10/10/80 Test
10/10/80 Tests are quick, short-term tests. Often times the test
results aren’t reusable. You’re looking for a quick boost on open
rates or click-thru by testing subject lines or senders.
● 10% get Subject Line A, 10% get Subject Line B, with the
winning subject line going to the remaining 80%
A/B Tests are 50/50 split tests where you’re looking for
statistically significant results that will be applied to future
campaigns.
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
7. II. Getting Your Boss On
Board
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
8. Before You Jump Into Testing
Getting stakeholder sign-off and support is
key!
LET’S TALK WITH A BOSS-BOSS ABOUT DIGITAL TESTING
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
● What’s exciting & nerve-wracking about digital testing?
● What can small/limited resource orgs do with digital testing?
● Three things needed to make the best case for digital
testing?
● What does Ann need in order to talk to higher-ups and other
stakeholders?
10. Let’s Dive In -- Email Testing
You can test just about everything
in email….
TEST TO IMPROVE OPEN RATES
● Subject Lines
● Senders
● Preview Language
● Time of Day
● Day of the Week
● Personalization
Subject Line Test Winner --
Increased click thru 45%
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
11. Let’s Dive In -- Email Testing
You can test just about everything
in email….
TEST TO IMPROVE CLICK-THRU RATES
● Design & Layout
○ Call-Out Box Placement, Images, Buttons,
Font-Size
● Copy
○ Ledes
○ Long vs. Short Copy
○ Topics
○ Personalized vs. Generic Language
Image Design Test --
Highlighting Reduced
Price increased donor
rate by 85%
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
12. Let’s Dive In -- Email Testing
You can test just about everything in
email….
TEST TO IMPROVE CLICK-THRU RATES
● Giving Options
○ Ask Language
○ Ask Amounts
○ Ask Types (1x vs. recurring)
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
Tested “Monthly Donor”
language vs. “Partner” language
-- results not statistically
significant (not enough results).
On list to re-test in future
campaigns
13. How Email Tests Work
Let’s dive into Engaging Networks:
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
14. How Email Tests Work
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
15. How Email Tests Work
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
16. How Email Tests Work
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
17. How Email Tests Work
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
18. IV. Donation Page Testing
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
19. How Donation Page Tests Work
The opportunities are endless:
TEST TO IMPROVE PAGE COMPLETION, DONOR RETENTION, AND AVERAGE GIFT:
● Layouts/Overall Design
● Single-step vs. Multi-step
● Images
● # of form fields
● Copy
● Pressure Points
○ “The Nudge” -- most people are giving $100 right now
○ Pop-up asks AFTER someone donates to upgrade their donation
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
20. How Donation Page Tests Work
The opportunities are endless:
TEST TO IMPROVE PAGE COMPLETION, DONOR RETENTION, AND AVERAGE GIFT:
● Ask string orientation (ascending vs. descending, horizontal vs. vertical)
● # of ask amounts
● Static vs. dynamic ask strings
● Include dollar signs or decimals or not
● Type of default ask (1x, recurring, quick-give)
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
21. How Donation Page Tests Work
Step 1: Make your donation pages
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
22. How Donation Page Tests Work
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
23. How Donation Page Tests Work
Step 3: Create a Split Test Page (takes less than 2 minutes)
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
24. How Donation Page Tests Work
Step 4: Connect the donation pages to the test
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
25. How Donation Page Tests Work
Step 5: Use the split test URL (1 URL randomizes visits to all donation
pages associated with the test)
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
26. How Donation Page Tests Work
Step 6: Analyze the results!
#ENCCDC19 #TestRinseRepeat @HRC @LautmanDC
Who’s testing every campaign? Testing once a year? Really wants to start testing.
What are the obstacles people are facing with testing?
Risk aversion -- can afford to sacrifice revenue on a test
A/B Split Test -- 50/50 through a whole campaign --
10/10/80 Test -- Subject Line (20% of list is test bed, 80% get the winner, so 90% will get the winning version)
Add Resources (non-Engaging Networks) --
-- Which Test Won
-- How to watch other non-profit emails and commercial emails to find testing ideas.
Highlight for both speakers how long they’ve been working with Engaging Networks (over 4 years each)
After agenda breakdown -- ask what people are currently testing; what they’re thinking about testing, where the roadblocks are with their testing plans.
Pause for questions at the end
Ann’s Prompt-Questions
n As a “boss” or digital program stakeholder, how do you feel about digital testing? What makes you most excited & most nervous about new tests being proposed?
n Now HRC is a huge organization – not everyone in this room has the type of resources, both budget & staff, to do digital testing regularly. For everyone here – I’m going to ask you to think back to 2009-2010, when HRC’s digital was a fraction of what it is now and there wasn’t as much budget or staff power as you have now – what are some of things we did back then, that are still applicable now?
o Lindsey’s example – find your smart interns who are wanting to show-off their skills. HRC’s first Facebook app was built and ran by an intern and that Facebook app set-off a massive wave of attention and change for HRC’s digital program – 100s of thousands of new people recruited, without any expense – just intern staff time.
n When your team or your consultants come to you with testing ideas, what are the three things they need to include in their proposals in order to make the best case – to help you understand the goal and impact of the test?
o This is where you talk about the initial investment and the long-term ROI, for example!
n Once these folks in here get their boss’s on their side and passionate about digital testing, they’re going to need their boss to get other stakeholders supporting digital testing. What information or documentation do you need Ann, as a boss, in order to talk to other bosses/stakeholders – like the Executive Director?