A/B testing is a fundamental skill for growing all aspects of a business. In this talk delivered at MozCon 2016, I lay out a path for evolving from generating a/b testing ideas to hypotheses; from executing tests to automating the process; and from working with a team to encouraging advocates.
7. 27%
3% 70%
70% OF MARKETERS
SAID CONVERSION
OPTIMIZATION IS
MORE OF A PRIORITY
FOR 2016.
Source: ConversionXL
SAME
LESS MORE
GREAT NEWS
@CaraHarshman #MozCon
8. 41%
56%
4%56% OF TEAMS WILL
ALLOCATE MORE
BUDGET TO
CONVERSION
OPTIMIZATION IN 2016.
Source: ConversionXL
SAME
MORE
LESS
DECENT NEWS
@CaraHarshman #MozCon
17. HYPOTHESIS
HILLARY CLINTON CAMPAIGN
@CaraHarshman #MozCon@CaraHarshman #MozCon
IF WE MAKE IT AS EASY AS POSSIBLE TO
SAVE YOUR CREDIT CARD INFO, THEN MORE
PEOPLE WILL DONATE, BECAUSE WE’VE
REMOVED THE FRICTION OF PAYMENT.
24. IDEAS HYPOTHESES
“LET’S…
ADD THIS
MOVE THAT
SWAP THIS FOR THAT
SHORTEN THIS
CHANGE THIS”
@CaraHarshman #MozCon
“IF WE DO THIS THEN THIS
WILL HAPPEN BECAUSE OF
THIS REASON.”
30. TOP ANALYTICS SOURCES
@CaraHarshman #MozCon
1. FUNNEL REPORT (GA)
2. HEAT MAP ON HOME & HIGH INTENT PAGES
3. FORM ERROR SUBMISSION REPORT (GA)
4. EXIT / PAGE BOUNCE REPORT (GA)
5. HIGH TRAFFIC / LOW CONVERSION LANDING PAGES
31. VOICE OF THE CUSTOMER
@CaraHarshman #MozCon
EMPATHY BASED ANALYSIS,
UNDERSTAND THE NEEDS OF THE
END USER AS A HUMAN
32. VOICE OF THE CUSTOMER LISTENING CHANNELS
@CaraHarshman #MozCon
1. CUSTOMER SUCCESS TEAM
2. RECORDED USER TESTING SESSIONS
3. COFFEE SHOPS
4. ON-PAGE SURVEYS
5. EMAIL SURVEYS
33. VOICE OF THE “HUMAN”
@CaraHarshman #MozCon
1. LIFT MODEL BY WIDER FUNNEL
2. CONVERSION EQUATION FROM OLI GARDNER
3. PSYCHOLOGY
4. DESIGN PRINCIPLES
5. NEUROMARKETING
50. PRIORITIZATION CHART: EASE VS. IMPACT
@CaraHarshman #MozCon
IF YOU CAN
MAKE TIME
DEPRIORITIZE
NEEDS A VERY
SOLID
HYPOTHESIS
DO IT!
EASE
IMPACT
51. PRIORITIZATION SCORE CARD
@CaraHarshman #MozCon
RULE 1 point 0 points
page part of funnel
outside
funnel
audience
100% of
visitors
specific
audience
success
metric
primary goal
secondary
goal
A RULE & POINT
BASED SYSTEM
WHERE EVERY
HYPOTHESIS GETS
A SCORE
52. “IF YOU COME TO ME WITH AN
IDEA AND IT’S NOT LIVE IN 2
WEEKS, IT’S NOT BECAUSE IT’S A
BAD IDEA – IT’S BECAUSE I HAVE
BETTER THINGS TO TEST.”
-Pauline, PM at Hotwire
70. 30%
29%
20%
21%
WHO DOES
OPTIMIZATION AT YOUR
ORGANIZATION?
Source: ConversionXL
(PRETTY) GOOD NEWS
@CaraHarshman #MozCon
NOBODY
A TEAM OF
PEOPLE
ONE PERSON PEOPLE
ACROSS
TEAMS
71. TOP 5 SKILL SETS TO HAVE
@CaraHarshman #MozCon
1. PROJECT MANAGER
2. DEVELOPER
3. DESIGNER
4. ANALYST
5. FEARLESSNESS
72. 2 TEAM EXAMPLES
@CaraHarshman #MozCon
JOSEPHINE AT HOTWIRE
DESIGN DEVELOPER
CUSTOMER
SUCCESS
PRODUCT OWNER,
HOTELS
PRODUCT OWNER,
FLIGHTS
CRAIG
HEAD OF GROWTH
JOSEPHINE
PRODUCT MANAGER
FOR TESTING
CRAIG AT EARNEST