This presentation is from Affiliate Summit West 2016 (January 10-12, 2016 in Las Vegas, NV). Session description: CRO is about research and process, not tactics and hacks. In this session, you’ll learn the tools and what it takes to evaluate campaigns & come up with clever solutions that impact the bottom line.

1. APPLYING OPTIMIZATION
FUNDAMENTALS
F O R M A J O R G A I N S

2. …OR 6 TIPS TO
SUPERCHARGE
Y O U R O P T I M I Z A T I O N P R O G R A M

3. TIP 1:
FOCUS ON
PROCESS…
NOT
‘HACKS’
@jtrondeau

4. @jtrondeau
WHAT WE MUST
REALLY?

6. Results:
No Statistical Difference
Confidence:
N/A
Takeaway:
The extra field increased
lead quality - so keep it!

8. Results:
31.3% Lift In Opt-Ins
Confidence:
95%

9. Source: Unbounce

10. @jtrondeau
WHAT WE
NEED:
PROCE
SS

11. TIP 2:
MEASURE
MULTIPLE
METRICS
@jtrondeau
(THAT MATTER)

12. @jtrondeau
It all starts with identifying goals &
KPIs!

13. @jtrondeau
Top
Funnel
Mid
Funnel
Bottom
Funnel
Engagement
metrics, not
directly
connected with
business goals.
Robust metrics,
directly connected
to business goals.
Generally
associated with
lead generation or
‘relationship’
changing metrics.

14. @jtrondeau
Page Goals Business
Goals
Immediate
Conversion
Clicks
Opt-Ins
RPV
Sales
AOV

15. @jtrondeau

16. @jtrondeau
Protip: Just because a top funnel
metric increased at x% doesn’t mean
a deep funnel metric will increase at
that rate (or increase at all).

17. @jtrondeau

18. @jtrondeau
Results:
2.01% Drop in Product Views
4.43% Drop in Add-to-Carts
10.59% Drop in Orders

19. BENEFITS OF
TESTING
MULTIPLE
METRICS
@jtrondeau
#1 GET THE FULL STORY
MORE USEFUL METRICS, MORE INSIGHTS
#2 CAN GET INSIGHTS ON LOW SAMPLES
IF THERE IS CONSISTENCY BETWEEN
PAGES/METRICS YOU MIGHT BE ABLE TO
CALL A TEST EARLIER
#3 BETTER ITERATIONS
TESTING IS AN ITERATIVE PROCESS, THE
MORE DATA YOU HAVE THE MORE YOU
CAN IMPROVE.

20. @jtrondeau
NO STATISTICAL DIFFERENCE

21. @jtrondeau

22. @jtrondeau

23. @jtrondeau

24. WHAT CAN I
LEARN?
@jtrondeau
DESPITE THE SMALLER SAMPLE SIZE FOR
THE DEEPER FUNNEL CONVERSION…
…THE CONSISTENCY IN ACTIONS OVER 3
PAGES AND 8 DAYS MADE US CONFIDENT
THAT THIS TEST WAS COMPLETE.

25. TIP 3:
ANALYTICS
TO IDENTIFY
PAGES
@jtrondeau

26. @jtrondeau

27. HOME CATEGORY PRODUCT CART THANK YOU
EVALUATING YOUR
FUNNEL WILL
HIGHLIGHT YOUR
PROBLEM AREAS
DETAILED FUNNEL ANALYSIS IS YOUR
TEST’S BACKBONE

28. 0
25
50
75
100
125
HOME CATEGORY PRODUCT CART THANK YOU
UniqueViews(In1,000’s)
Page Viewed

30. PROTIP
@jtrondeau
WE CAN SURMISE A LOT FROM
QUANTITATIVE DATA…
…WE DEVELOP HYPOTHESES WHEN WE
EVALUATE QUALITATIVE DATA.
(WE’LL TALK ABOUT THIS LATER)

31. @jtrondeau

32. PICK PAGES
THAT
MATTER
@jtrondeau
#1 HIGH VALUE PAGES WITH A HIGH
BOUNCE RATE
BOUNCES ARE BAD NEWS BEARS
#2 DEEP FUNNEL PAGES
PEOPLE JUMPED THROUGH YOUR HOOPS,
BRING THEM HOME
#3 FUNNEL VALLEYS
CHECK THE PREVIOUS PAGE TO FIND OUT
WHY NO ONE IS MOVING DOWN THE
FUNNEL

33. OR MY
PERSONAL
FAVORITE
@jtrondeau
High Visit, High Bounced Pages

34. TIP 4:
FIX WHAT’S
BROKEN…
@jtrondeau

35. TIP 4:
…AND TEST
WHAT’S
AMBIGUOUS
@jtrondeau

36. @jtrondeau

37. @jtrondeau
Just Fix It
Test/Optimize

38. TIP 5:
SCHEDULE A
FINITE STOP
TIME
@jtrondeau

39. @jtrondeau
LIFT LOSS NULL

40. @jtrondeau

41. @jtrondeau
Results:
6.49% Increase In Sales
Confidence:
95%

42. @jtrondeau

43. @jtrondeau
Results:
Decreased Add To Cart
Clicks by 6.6% For New
Visitors.
Confidence:
95%

44. @jtrondeau
NO STATISTICAL DIFFERENCE

45. @jtrondeau
LIFT LOSS

46. @jtrondeau
“Netflix considers 90% of what they try to be wrong” -- Mike Moran
“80% of the time you/we are wrong about what a customer wants”
-- Avinash Kaushik
Microsoft’s statistic from thousands of controlled experiments:
only 10-30% of experiments move the metrics they were
designed to improve”
— Ronny Kohavi

47. @jtrondeau
FACTORS THAT
IMPACT
TEST
TIMING
#1 LIFT PERCENTAGE
WHAT WAS THE LIFT OR LOSS
#2 CONFIDENCE RATE
95% OR BETTER
#3 NUMBER OF VARIANTS
2 OR MORE
#4 NUMBER OF VARIANTS
TESTED POPULATION PER DAY

48. @jtrondeau
WORK
BACKWARD
#1 PICK TIME PERIOD
1-6 WEEKS
#2 RECORD UNIQUE TRAFFIC
WHAT’S THE TRAFFIC FOR THE TIME
PERIOD
#3 NUMBER OF VARIANTS
DIVIDE TRAFFIC BY VARIANTS
#4 DESIRED CONFIDENCE
95%
#4 CALCULATE LIFT PERCENTAGE
CAN YOU GET X LIFT? IF YES, DO IT!

49. @jtrondeau
IF YES,
THEN
SCHEDULE(AND STICK TO IT!)

50. TIP 6:
FOCUS ON
THE USER
@jtrondeau

51. CONVERSION RATE OPTIMIZATION ISN’T ABOUT
INCREASING CONVERSION
IT’S ABOUT CREATING MEETING USER
EXPECTATIONS

52. WE NEED
DETAILED
USER
RESEARCH
@jtrondeau
#1 HEATMAPS
SEE WHERE PEOPLE ARE
CLICKING/SCROLLING
#2 USER SURVEYS
GET INFORMATION FROM YOUR BEST
USERS
#3 SESSION RECORDINGS
WATCH EXACTLY HOW PEOPLE INTERACT
ON YOUR SITE
#4 USABILITY TESTS
GET SITE TESTERS AND ASK FOR
FEEDBACK

53. QUALITATIVE
DATA =
COMPETITIVE
EDGE
@jtrondeau

54. @jtrondeau
EXAMPLE OF A
HEATMAP

55. @jtrondeau
EXAMPLE OF A
SCROLL
MAP

56. @jtrondeau
HOW A
HEATMAP
INCREASED
SALES

57. @jtrondeau
We Saw a 40.9%
Increase In DM Lab Sales
When We Added A Link
To Our Sales Page

58. @jtrondeau
EXAMPLE OF A
SESSION
RECORDING

59. @jtrondeau
LESSONS
LEARNED

60. @jtrondeau
EXAMPLE OF A
SURVEY

61. @jtrondeau
SOME
SWEET
SURVEY
TIPS
#1 ASK A SINGLE QUESTION
IF ASKING FOR MORE, THEN
SET TIME CONSTRAINTS
#2 OPEN ENDED = POWER
YOU’LL GET LESS RESPONSES,
BUT BETTER DATA
#3 KNOW IT’S A SEGMENT
THE TYPE OF PEOPLE WHO
ANSWER SURVEYS AREN’T AN
ACCURATE REPRESENTATION
OF THE WHOLE

62. @jtrondeau
• Was it was easy to find what you were looking for?
• Were the words/vocab used to define categories/sub-categories
clear to you?
• Do you have any suggestions to improve our website navigation?
• Does our website look credible to you?
• Is our pricing clear?
• Is there anything else you’d like to know before signing up with us?
• Will you shop with us again? Why/why not?
• Do you think the form has any confusing/unnecessary input fields?

63. @jtrondeau
EXAMPLE OF A
USER TEST

64. @jtrondeau
MY FAVORITE
TYPE OF
QUAL.
DATA
COMMON CUSTOMER SERVICE
QUESTIONS AND
COMMON SALES QUESTIONS

65. @jtrondeau
QUICK
RECAP
#1 CLEAN YOUR DATA
MAKE SURE YOU TRUST WHAT’S IN GA
#2 GET A TESTING TECH
VWO, OPTIMIZELY, ETC…
#3 GET A QUALITATIVE SOURCE
HEATMAPS, SURVEYS, CUSTOMER DATA,
ETC…
#4 I.D. PROBLEM PAGES & WHY
#5 RUN TEST & REPORT

66. @jtrondeau
• Email: justin.rondeau@digitalmarketer.com
• Twitter: @jtrondeau
• Website: digitalmarketer.com
Justin Rondeau
Director of Optimization,
Digital Marketer
THANKS! QUESTIONS?