Google is a black box, and for almost 20 years SEOs have run experiments and tested ideas trying to understand what makes the search engine tick. Until recently, it's been really hard to run robust tests that isolate the effects of SEO changes. At Distilled, we have been using new tools and statistical approaches to run split-tests. In this session, Tom is going to talk about how you can run your own A/B tests, some of the experiments we've run and the results we've seen, and share some thoughts about the future of SEO testing.

1. SEO Split-Testing
Why you should. How you can. What we’ve learnt.
@TomAnthonySEO

4. But I wore the juice!
McArthur Wheeler

5. ?
Do you think lemon
juice makes you
invisible?

6. Do you think HTTP is
better than HTTPS?

7. Are multiple
schema.org entries a
good idea?

8. Are 302’s really as
good as 301’s?

9. Be careful with
untested recommendations.
Don’t blindly
believe Google.
!

10. So today I want to discuss 3 things
How good are we at ranking factors?
How to run tests & pitfalls to avoid
Some of the tests we’ve been running

11. RANKING FACTORS

12. Search: future of tv advertising
http://adage.com/article/media/future-tv/303508/

13. Search: future of tv advertising
http://contentmarketinginstitute.com/2016/04/television-advertising-native/

14. Search: future of tv advertising
DA PA LRDs Rank
http://dis.tl/rankq-1a
91 47 2 ?
http://dis.tl/rankq-1b
86 57 9 ?

15. Search: future of tv advertising
DA PA LRDs Rank
http://dis.tl/rankq-1a
91 47 2 3
http://dis.tl/rankq-1b
86 57 9 8

16. https://www.jensinkler.com/fix-your-kettlebell-swing/
Search: kettlebell technique

17. http://www.mensfitness.com/training/pro-tips/perfect-kettlebell-swing
Search: kettlebell technique

18. Search: kettlebell technique
DA PA LRDs Rank
http://dis.tl/rankq-2a
41 42 12 ?
http://dis.tl/rankq-2b
76 44 20 ?

19. Search: kettlebell technique
DA PA LRDs Rank
http://dis.tl/rankq-2a
41 42 12 4
http://dis.tl/rankq-2b
76 44 20 6

20. http://www.macworld.co.uk/how-to/iphone/how-to-back-up-iphone-ipad-backup-3595444/
Search: do I need to backup my iphone

21. http://www.idownloadblog.com/2013/08/28/not-enough-icloud-storage-iphone-cannot-be-backed-up/
Search: do I need to backup my iphone

22. Search: do I need to backup my iphone
DA PA LRDs Rank
http://dis.tl/rankq-3a
77 59 16 ?
http://dis.tl/rankq-3b
67 34 3 ?

23. Search: do I need to backup my iphone
DA PA LRDs Rank
http://dis.tl/rankq-3a
77 59 16 7
http://dis.tl/rankq-3b
67 34 3 5

24. How did we do
compared to others?

25. We surveyed 2000 people for 10 queries

26. % of UK search result pairs predicted correctly
30%
40%
50%
60%
70%
Laypeople

27. % of UK search result pairs predicted correctly
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO

28. % of UK search result pairs predicted correctly
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO

29. % of UK search result pairs predicted correctly
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip

30. Surprisingly hard.

31. DeepRank

33. % of UK search result pairs predicted correctly
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip

34. % of UK search result pairs predicted correctly
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip DeepRank

35. Our instincts aren’t great!
We need data and ways
to validate.

36. CRO Testing

37. All CRO teams run tests.
If they didn’t you wouldn’t
take them seriously.

38. Why is that not true for
SEO teams?

39. SEO TESTING

40. IMEC Labs
IMEC Labs
Run experiments
against the SERPs to
understand aspects
of Google’s algorithm.

41. Blog Post

42. Blog Post

43. You can do proper
testing too!

44. RUN YOUR OWN TESTS

45. Blog Post - http://dis.tl/mark-post

46. ‘Traditional’ CRO A/B Testing is User Oriented

47. SEO A/B Testing is Page Oriented

48. SEO A/B Testing includes Googlebot

49. Introduction to SEO A/B Testing
Blog Post

50. How Split-Testing Changes Consulting
Blog Post

51. 3 Steps to DIY SEO Split-Tests
1. Create two buckets of pages.
2. Make a change to all pages in one bucket.
3. Analyse which bucket performs better.

52. 1. Create two buckets of pages.
A good DIY approach is using GA segments. Easy to make (see link).
Suggestion: Create Segments by category (e.g. blog tag, product category).

53. 2. Make a change to all pages in one bucket.
Control Variant
This will be your variant bucket.
Utilise your CMS where possible. Ask Dev for scalable approach.

54. 3. Analyse which bucket performs better.
A naive approach can just compare absolute traffic, if you had a close match before.
But needs a big uplift and hard to create such Segments.
Better approach is using Google's Causal Impact library — read this great Lunametrics post.
Variant
Control

55. DistilledODN
Your Server Your Home Page
CDN
Distilled ODN
Your Optimized
Home Page

56. SOME TESTS WE’VE RUN

57. ConcertHotels.com

58. ConcertHotels.com: Test Setup
~20,000 location category pages
Pages
Title: <<Location>> Hotels, NY | ConcertHotels.com
H1: <<Location>> Hotels
Before (Control)
Title: Hotels near <<Location>>, NY | ConcertHotels.com
H1: Hotels near <<Location>>
After (Variant)

59. ConcertHotels.com: Results
2.5 weeks to get to significance.
Gradual improvement in organic performance leading to steady amount of higher traffic.
Results

60. SmokyMountains.com

61. SmokyMountains.com: Test Setup
~100 lodging pages
Pages
schema.org markup: @type “WebSite”. Generic on all pages.
Before (Control)
schema.org markup: @type “LodgingBusiness”. Customised to each page.
After (Variant)

62. SmokyMountains.com: Results
Fewer test pages than previous test, so it is less smooth but is detected much quicker.
Traffic uplift here is estimated to be ~5%.
Results

63. iCanvas.com

64. iCanvas.com: Test Setup
~3200 artist category pages
Pages
meta description: Shop our selection of canvas prints by Banksy, each hand-stretched over
museum-quality bars and printed with brilliant, fade-resistant inks. Free shipping and returns.
internal links: ~50 self referential links
Before (Control)
meta description: Banksy Prints on Canvas, including There Is Always Hope Balloon Girl, Life Is
Beautiful and others. Free shipping and returns.
internal links: <removed>
After (Variant)

65. iCanvas.com: Results
2 weeks to get significance.
Number of pages is between the previous tests: a few days to be noticed but quite smooth.
Results

66. SOME OTHER TEST IDEAS

67. Tabbed vs Flat

68. Breadcrumbs
!
!
!

69. Remove Interstitials

70. We removed all the ads from pages.
Immediate uplift. Ad revenue was 1%. Clear ROI.

71. Build a library of tests

72. POTENTIAL PITFALLS

73. You can’t assume traffic equality
between “buckets” of pages
This is why we build a counterfactual comparison using control pages.
Use Google’s Causal Impact library to do it yourself.

74. Pay attention to:
Amount of traffic
& number of pages
These two factors will determine how quickly you can test and what size uplifts you can detect.

75. New pages that
appear during tests
The simplest approach is to just ignore all new pages that didn’t exist before the test started.

76. Different pages can have different
seasonality
For example, “roses” pages on valentine’s day. You need to cut outliers.

77. You could damage conversions,
so pay attention to those metrics too.
Tag people in your analytics depending if they arrive on Control or Variant

78. Run Null Tests
Verify your test framework. Check for false positives.

79. RECAP

80. CROs are testing,
we should too! I love testing!

81. You make me so
happy, Tom
Boss Friendly!
Get Attribution!
Get Dev Time!

82. Small site?
Use Mark Edmondson Tool
Bigger site?
Use an A/B Approach

83. Thanks!
@TomAnthonySEO

84. www.distilledodn.com
DistilledODN allows you to test exactly which changes to your
website will result in an uplift in traffic from search engines.
Our new SEO A/B Testing platform is available.
@TomAnthonySEO

85. Image Credits
• Buckets - mamarazzi
• Rose - alicelingching
• Girders - JFB119
• Rubiks - Wallpaperzone