A testing playbook combines the best practices of testing and optimization, along with communication strategies, education, and gaining buy-in from your client. Andrew Richardson, Senior Director of Analytics at Delphic Digital, provides a peek behind the curtain to reveal how Delphic prioritizes tests, recruits/trains/staffs-up for a testing practice, and moved from A/B to multivariate testing. Come with and open mind, walk away with a Testing Playbook Template you can put to use at once.

2. I CAN’T EAT LUNCH
YET
THIS GUY KEEPS
TALKING…

3. This is your
BRAIN
An attention span of
8 seconds
*https://advertise.bingads.microsoft.com/en-us/insights @analyticandrew #eMetrics

5. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION. 
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics

6. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION. 
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics

8. 2.6%
2.8%
2.8%
3.2%
3.3%
3.4%
5.0%
5.0%
5.5%
6.0%
Higher Education
Real Estate
Health
Legal
Home Improvement
Business Services
Travel
Business Consulting
Credit & Lending
Vocational Studies & Job Training
*Unbounce Conversion Benchmark Report – March 2017
HOW DO YOU COMPARE?

9. Never stop
testing,
and your
advertising will
never stop
improving.
- David Ogilvy -

10. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION. 
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics

11. DAYLIGHT
Your ideas see

12. No more
HIPPO

13. 5xHigher ROI than
PPC
Up to

14. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION. 
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics

15. Not enough
TRAFFIC

17. We have
NO IDEAwhat to test

21. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION. 
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics

22. Nothing should be
OFF LIMITS

23. T-shirt
TEST
size your
ING

25. Testing is often about
CHANGINGpolicy & perception

26. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION. 
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics

27. Ok, I am
ALL INNow what?

28. Tool Selection:
BUILD or BUY

30. 7 TOOLS
MarTech 2011:

32. 175 TOOLS
MarTech 2017:

33. THE PEOPLE
The most important thing

34. THE PEOPLE
The most important thing
BUT!

35. THE PEOPLE
The most important thing
one: focus efforts where they matter most
two: avoid conflict-put opinions to the test
three: empower people to optimize & be data-drive
four: align teams around shared goals
five: collaborate around hypothesis generation,
test
ideas & execution

36. PROCESS
IS
CRUCIAL
FOR
SUCCESS

37. RESEARCH
1.
PRIORITIZE
2.
EXPERIMENT
3.
ANALYZE, LEARN,
REPEAT
4.

38. RESEARCH
1.
“If A/B tests are used in lieu of research, the variations
are essentially guesses. We can improve outcomes of
A/B tests by incorporating UX research to:
•improve cause identification
•develop more realistic hypotheses
•identify more opportunities for experimentation”
Nielsen Norman Group: www.nngroup.com

39. RESEARCH
1.

40. RESEARCH
1.

41. RESEARCH
1.

42. RESEARCH
1.

43. RESEARCH
1.

44. A theory can be
proved by
experiment; but no
path leads from
experiment to the
birth of a theory.

45. RESEARCH
1.
Issue
• What is the
conversion
problem?
Cause
Theory
• What do you
think is causing
the problem?
Variation
Hypotheses
• What do you
propose
changing?
Goals
• What outcomes
are expected?

46. RESEARCH
1.
Issue
• What is the
conversion
problem?
Cause
Theory
• What do you
think is causing
the problem?
Variation
Hypotheses
• What do you
propose
changing?
Goals
• What outcomes
are expected?

47. RESEARCH
1.

48. RESEARCH
1.
User-Experience to Improve Optimization Testing
• Defining user intent and objections
• Why did you visit? Were you successful? If not, why?
• Exposing interface flaws
• Changing colors vs. changing form fields
• Measuring findability
• Tree testing, site search, heatmaps
• Cleaning up design variations
• Reduce friction, maximize testing

49. RESEARCH
1.
Bug tracking Brainstorming Site Search
Web Analytics Site Surveys Chat

50. PRIORITIZE
2.
• TEST
• Where you put stuff for testing
• INSTRUMENT
• Fixing, adding, improving tagging, events, etc.
• HYPOTHESIZE
• Found something, but no clear solution
• JUST DO IT (JDI)
• No brainers
• INVESTIGATE
• Need to dig further, ask more questions
Craig Sullivan: @optimiseordie

51. PRIORITIZE
2.

52. PRIORITIZE
2.
Craig Sullivan Hypothesis Kit
one: Because we saw (data/feedback)
two: We expect that (change) will cause (impact)
three: We’ll measure this using (data metric)
https://medium.com/@optimiseordie/hypothesis-kit-2-eff0446e09fc

53. PRIORITIZE
2.
Craig Sullivan Advanced Kit
one: Because we saw (qual and quant data)
two: We expect that (change) for (population)
will cause (impact(s))
three: We except to see (data metric(s)
change) over a period of (x business
cycles)
https://medium.com/@optimiseordie/hypothesis-kit-2-eff0446e09fc

54. EXPERIMENT
3.

55. EXPERIMENT
3.

56. EXPERIMENT
3.

57. EXPERIMENT
3.

58. ANALYZE, LEARN,
REPEAT
4.
Insert random cycle chart here

59. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION. 
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics

60. • Headlines
• Sub headlines
• Paragraph Text
• Testimonials
• Call to Action text
• Call to Action Button
• Links
• Images
• Content near the
fold
• Social proof
• Media mentions
• Awards and badges

61. DESKTOP
Segment
From
MOBILE

62. SWEAT
Don’t
the small stuff

63. IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION. 
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics

64. audience.ask(questions);
@analyticandrew #eMetrics

65. Never stop
testing,
and your
advertising will
never stop
improving.
- David Ogilvy -