Showing the complexity of Google's search results, and the lack of understanding we generally have of what works and what doesn't - meaning we need to use a more scientific approach.
Finally - a bunch of lessons and data from split tests we have run
7. I was thinking about it like it was a
math puzzle and if I just thought
really hard it would all make sense.
-- Kevin Lacker (@lacker)
8. Hey why don't you take the square
root?
-- Amit Singhal according to Kevin Lacker (@lacker)
9. oh... am I allowed to write code that
doesn't make any sense?
-- Kevin Lacker (@lacker)
10. -- Amit Singhal according to Kevin Lacker (@lacker)
Multiply by 2 if it helps, add 5,
whatever, just make things work
and we can make it make sense
later.
37. Instead of comparing the performance of the control pages directly with the variant pages, we build a
forecast of what’s called the counterfactual which is an estimate of what would have happened if we hadn’t
made the change. We use the control group to make a counterfactual forecast that takes into account
seasonality and site-wide changes.
The black line on the chart above is the actual organic traffic to the variant pages. The blue line is the
counterfactual.
More: Distilled blog post and free forecasting tool
38. It’s easiest to analyse the results by looking at the cumulative difference over time between the actual
organic traffic and the counterfactual.
The pale blue area is the 95% confidence interval.
We can see a (statistically) zero effect for an initial time while Google crawls and indexes the test,
followed by steady growth. A couple of weeks in, the confidence interval goes above zero and we have a
winning test.
More: Distilled blog
39. It’s easiest to analyse the results by looking at the cumulative difference over time between the actual
organic traffic and the counterfactual.
The pale blue area is the 95% confidence interval.
We can see a (statistically) zero effect for an initial time while Google crawls and indexes the test,
followed by steady growth. A couple of weeks in, the confidence interval goes above zero and we have a
winning test.
More: Distilled blog
Hashtag winning
40. Further reading for those interested:
● Predicting the present with Bayesian structural time series [PDF]
● Inferring causal impact using Bayesian structural time series [PDF]
● CausalImpact R package
● Finding the ROI of title tag changes
More: Distilled blog
41. 1. Adding structured data
2. Adding ALT attributes
3. Setting exact match title tags
4. Using JS to show content
5. Removing SEO category text
42. Credit to my colleague Dom who
runs our split-testing projects
@dom_woodman
43. 1. Adding structured data
2. Adding ALT attributes
3. Setting exact match title tags
4. Using JS to show content
5. Removing SEO category text
50. 1. Adding structured data
2. Adding ALT attributes
3. Setting exact match title tags
4. Using JS to show content
5. Removing SEO category text
51. Title tag before: Which TV should I buy? - Argos
Title tag after: Which TV to buy? - Argos
What happens when you match title tags to the greatest search volume?
62. This is why we have been investing so much in split-testing
Check out odn.distilled.net if you haven’t already. The team will be happy to
demo for you.
We served ~5 billion requests last quarter and recently published
everything from response times to our +£100k / month split test.
63. But I’m also seeing more subtle impacts on my recommendations:
● You can recommend small tweaks and see the benefits compound
● You can test wild hypotheses with unknown upsides
● You can try things that might have a downside (more focused targeting, less copy, etc.)
And that’s even before you get the benefits of testing clickthrough rate, and the benefits of pretty charts
to show the boss highlighting the impact of your work!
More: blog post