1. How one company implements an entire testing methodology
every day
The Nuts and Bolts
Ryan Hutchings
Director of Marketing
VacationRoost
Daniel Burstein
Director of Editorial Content
MECLABS

2. Session speaker
2
Ryan Hutchings
Director of Marketing
VacationRoost
Hutchings has more than 10 years of online marketing experience. He leads the
marketing team for VacationRoost, a $120 million private company in the travel
industry. From advertising to analytics, he organizes a comprehensive and robust
online marketing strategy focused on tracking and optimization. His specialties
include SEO, PPC, testing, conversion rate optimization, advanced data modeling,
analytics and every technical aspect related to online marketing.
Hutchings is a Google AdWords certified professional, SEOmoz Pro member,
nationally recognized industry speaker and participant of top online marketing
communities. He also help found the SLCSEM organization, which is Utah’s largest
nonprofit group of online marketing professionals.
@HutchSEO

3. About VacationRoost

4. About VacationRoost
• 25 websites
• Always running 2 to 3 tests
• 100+ tests run across landing pages, homepages,
lead forms and shopping carts
• Goal of 10% incremental lift in total revenue
generated by testing

6. Fundamental
Question:
How can marketers systematically run
tests to achieve consistent lifts?
and hundreds of other tests….

7. An overall testing process …
Build

8. … to support an iterative optimization cycle
Collecting
data
Planning
experiments
Preparing
treatments
Conducting
tests
Analyzing
results

9. The dual testing track
A/B split test
• Internal system with server-side
splitting, custom variables
• Full manual reporting –
spreadsheets and database
reports
Largetest
Search results
Cart
Products
Testing
Landing page optimization
• Third-party software
• Internal analysis determines
what to test
Smalltest PPC and SEO landing pages
Headlines/CTAs
Images

10. The dual testing track
A/B split test
• Internal system with server-side
splitting, custom variables
• Full manual reporting –
spreadsheets and database
reports
Largetest
Search results
Cart
Products
Testing
Landing page optimization
• Third-party software
• Internal analysis determines
what to test
Smalltest PPC and SEO landing pages
Headlines/CTAs
Images

11. How to test small
Decide Identify Hypothesize Design Analyze
Decide what to test

12. 90%
Replace “our recommendations” section with content snippet
for value proposition banner usage
Restyle “most popular destinations” change hierarchy 40%

13. How to test small
Decide Identify Hypothesize Design Analyze
Identify the target conversion goal

14. What’s a good goal for bounce rates?
> 50%
40%-50% 30%-40%
< 30%

15. What’s a good goal for CTR to the next page
or step?
> 60%
50%-60%30%-50%
< 30%

16. How to test small
Decide Identify Hypothesize Design Analyze
Create a hypothesis

17. Small Test: PPC landing pages
17

18. Small Test: PPC landing pages
18

19. Small Test: PPC landing pages
19
How can we
reduce friction and
anxiety on this
page?

20. How to test small
Decide Identify Hypothesize Design Analyze
Wireframe, design and launch treatment

21. Small Test: PPC landing pages
21

22. Small Test: PPC landing pages
22

23. Small Test: PPC landing pages
23

24. Small Test: PPC landing pages
24

25. How to test small
Decide Identify Hypothesize Design Analyze
Analyze results and calculate ROI

26. Small Test: PPC landing pages
26
427% increase in CTR

27. Small test validation tool
95.2%

28. ROI template
Let’s take a
closer look…

29. ROI template
Average ticket
Jan
Feb
Mar.
Apr
May
Jun
Jul.
Aug
Sept
Oct
Nov.
Dec
Close rate
Jan
Feb
Mar.
Apr
May
Jun
Jul.
Aug
Sept
Oct
Nov.
Dec
Lead
conversion
Jan
Feb
Mar.
Apr
May
Jun
Jul.
Aug
Sept
Oct
Nov.
Dec
Contribution
% of sales

30. ROI template
Incremental Increases
New Bounce Rate
Leads
Sales $
Contribution $
53
$ 40,307
$ 3,225
Incremental Increases
New Bounce Rate
Leads
Sales $
Contribution $
20%
7
5,246 $
420 $
At 10%
At 30%
158
$ 119,256
$ 9,541
25 %
2 d
1,749 $
140 $

31. How do you get the resources to run
large tests?

32. The dual testing track
A/B split test
• Internal system with server-side
splitting, custom variables
• Full manual reporting –
spreadsheets and database
reports
Largetest
Search results
Cart
Products
Testing
Landing page optimization
• Third-party software
• Internal analysis determines
what to test
Smalltest PPC and SEO landing pages
Headlines/CTAs
Images

33. How to test big
Decide Collaborate Identify Create Design Analyze
Decide on an element to optimize

34. Looking at the same
elements as the small
tests.

35. The basic funnel
Landing Pages
Search Results
Property
Cart
Billing
Purchase
3.5%
1.7%
“Rising tide”
A/B
(large test)
Single page LPO
(small test)
40%
1.7%
Purchase

36. Determining a “good” conversion rate
Source: MarketingSherpa.com/ecommerce

37. How to test big
Decide Collaborate Identify Create Design Analyze
Work with IT to define variables or events

38. Large Testing: Variables are the key
38
Conditions
Segment your users and/or their visits to single or multi-visit conditions.
inlineLeadForm1
mrSearchPage1
Notest
simpleModalLeadForm
simpleModalLeadForm4
control
test

39. How to test big
Decide Collaborate Identify Create Design Analyze
Identify target conversion goal for control

40. The basic funnel
Landing Pages
Search Results
Property
Cart
Billing
Book now, Web lead,
interactions/events
Checkout, add “cross
sell”
Book, fill out info
Bounce
rates, CTR
Purchase
“Rising tide”
A/B
(large test)
Single page LPO
(small test)

41. The basic funnel
Landing Pages
Search Results
Property
Cart
Billing
Book Now, web lead,
interactions/events
Checkout, add “cross
sell”
Book, fill out info
Bounce
rates, CTR
Purchase
Book Now, web lead,
interactions/events
“Rising tide”
A/B
(large test)
Single page LPO
(small test)
10% - 30% minimum targeted gains

42. Conversion goal results

43. How to test big
Decide Collaborate Identify Create Design Analyze
Create hypothesis and set up validation sheet

44. Manual testing database
Number of Days
40

45. Control Treatment
Large Test: Lead forms 1
Security seals

46. Control
Fail – no lift.
Large Test: Lead forms 1
Treatment

47. Control Treatment
Large Test: Lead forms 2
Shorter form

48. Control
Fail – no lift.
Large Test: Lead forms 2
Treatment

49. Control Treatment
Large Test: Lead forms 3
Option to call

50. Control Treatment
Fail – no lift.
Large Test: Lead forms 3

51. How to test big
Decide Collaborate Identify Create Design Analyze
Wireframe, design and launch treatment

52. Large Test: Lead forms

53. Large Test: Lead forms

54. Large Test: Lead forms
19% Lift

55. How to test big
Decide Collaborate Identify Create Design Analyze
Analyze results and calculate ROI

57. Manual testing database

58. Testing roadmap – it’s really:
Search Results
Property Details
Cart
Billing info
Booking

59. ROI template
Same as a small test.

60. Setting up your own programmatic testing
1
2
3
4
5
Custom variable tracking system, spreadsheet templates are vital building blocks
Dedicate a person and tie performance to testing
“Top of the funnel” to the “bottom conversion” may have too much noise in
between – keep it simple and isolated
Build a reporting system that ties front end and back end together
Get buy-in from the management team

61. Questions?

62. Thank you
Ryan Hutchings
Marketing Director
VacationRoost
@HutchSEO
Daniel Burstein
Director of Editorial Content
MECLABS
@DanielBurstein
Black & White
Headshot
Black & White
Headshot