We think we know how searchbots interact with our websites, but do we really? What data is there that tell us where they visited, what they requested, how long they spent? This talk examines various data sources, visualises searchbots visits (animating them over time), explores how we can help searchbots - and even attempts to turn their visits into sound.

1. Searchbots:
Lost Children ... or ... Hungry Psychopaths?
What Do Searchbots Actually Do? (and why it matters)
© 2007-2012 Roland Dunn

2. © 2007-2012 Roland Dunn

3. “Googlebot ... may be unable to completely index
all the content on your site”
© 2007-2012 Roland Dunn

4. © 2007-2012 Roland Dunn

5. © 2007-2012 Roland Dunn

6. © 2007-2012 Roland Dunn

7. © 2007-2012 Roland Dunn

8. Doesn’t tell us what searchbots index
© 2007-2012 Roland Dunn

9. © 2007-2012 Roland Dunn

10. © 2007-2012 Roland Dunn

11. © 2007-2012 Roland Dunn

12. © 2007-2012 Roland Dunn

13. Webserver Logfiles ....
© 2007-2012 Roland Dunn

14. Webserver Logfiles ....
© 2007-2012 Roland Dunn

15. Webserver Logfiles ....
© 2007-2012 Roland Dunn

16. © 2007-2012 Roland Dunn

17. Over 2 months, 40% of all Googlebot’s requests
were to …. just 2 URL’s (on-site search URLs)
© 2007-2012 Roland Dunn

18. Over 2 months, 40% of all Googlebot’s requests
were to …. just 2 URL’s (on-site search URLs)
On a website with approximately 40,000 URLs in
Google’s index (allegedly, according to site:…)
© 2007-2012 Roland Dunn

19. Over 2 months, 40% of all Googlebot’s requests
were to …. just 2 URL’s (on-site search URLs)
On a website with approximately 40,000 URLs in
Google’s index (allegedly, according to site:…)
On a website serving approx. 150-200K unique
natural search visits/month
© 2007-2012 Roland Dunn

20. Over 2 months, 40% of all Googlebot’s requests
were to …. 2 URL’s.
On a website with approximately 40,000 URLs in
Google’s index.
On a website serving approx. 150-200K unique
natural search visits/month
© 2007-2012 Roland Dunn

21. © 2007-2012 Roland Dunn

22. Categorise URLs by “top level section” – perhaps more
useful than just URLs ...
© 2007-2012 Roland Dunn

23. Table of “top level sections” requested by Googlebot
© 2007-2012 Roland Dunn

24. Chart of “top level sections” requested by Gbot – 6 months
© 2007-2012 Roland Dunn

25. © 2007-2012 Roland Dunn

26. Does Googlebot request all URLs served by a website?
© 2007-2012 Roland Dunn

27. Does Googlebot request all URLs served by a website?
No!
© 2007-2012 Roland Dunn

28. How Does Googlebot Spend Its Time? What Does it Request?
© 2007-2012 Roland Dunn

29. Does Googlebot spend its time cost-effectively?
© 2007-2012 Roland Dunn

30. Does Googlebot spend its time cost-effectively?
Not always.
© 2007-2012 Roland Dunn

31. Googlebot does not always request all content
© 2007-2012 Roland Dunn

32. Googlebot can become distracted, obsessed, or even lost:
© 2007-2012 Roland Dunn

33. Googlebot can become distracted, obsessed, or even lost:
• On-site search
• Additive filters/faceted URLs e.g.
“shoes?size=3&colour=green&price=10&brand=smith”
• Sections with thin and/or very similar content
© 2007-2012 Roland Dunn

34. Googlebot may not spend its time efficiently
It needs help focusing on what we value
© 2007-2012 Roland Dunn

35. © 2007-2012 Roland Dunn

36. SMX West, March 2011, Matt Cutts:
(http://goo.gl/ZZz7E):
“... if Google determines a site isn’t as useful to
users, they may not crawl it as frequently"
© 2007-2012 Roland Dunn

37. © 2007-2012 Roland Dunn

38. Conclusions:
• Searchbots do not always request all content
• Searchbots can become distracted
• Searchbots may use their time inefficiently
• Their visits are precious – we need to treasure them
• We may need to help them focus
© 2007-2012 Roland Dunn

39. Suggestions:
• Embrace logfiles – full of useful information
© 2007-2012 Roland Dunn

40. Suggestions:
• Embrace logfiles – full of useful information
• Check searchbot behaviour
© 2007-2012 Roland Dunn

41. Suggestions:
• Embrace logfiles – full of useful information
• Check searchbot behaviour
• If distracted, lost, inefficient, poor experience:
• Alter internal navigation & linking (e.g. flatten hierarchy)
• Robots.txt out (blunt approach)
• Alter URL construction (expensive!)
© 2007-2012 Roland Dunn

42. © 2007-2012 Roland Dunn

43. Lost Children ... or ... Hungry Psychopaths?
© 2007-2012 Roland Dunn

44. Or .... Distracted Teenager?
© 2007-2012 Roland Dunn

45. © 2007-2012 Roland Dunn

46. Sonification: What do Search Engines Requests Sound Like?
© 2007-2012 Roland Dunn

47. Sonification: What do Search Engines Requests Sound Like?
© 2007-2012 Roland Dunn

48. © 2007-2012 Roland Dunn

49. Online Branding: Natural Search:
http://www.refinedpractice.com/ http://www.cloudshapes.co.uk/
T: @RefinedPractice T: @roland_dunn
Slides Available At:
http://www.cloudshapes.co.uk/talks/
© 2007-2012 Roland Dunn