1. ionSearch 2012
Roland Dunn
Partner – Refined Practice

2. Searchbots:
Lost Children ... or ... Hungry Psychopaths?
What Do Searchbots Actually Do? (and why it matters)
© 2007-2012 Roland Dunn

3. © 2007-2012 Roland Dunn

4. Straightforward for humans and searchbots to find your content?
© 2007-2012 Roland Dunn

5. Do you provide what your potential customers want?
Straightforward for humans and searchbots to find your content?
© 2007-2012 Roland Dunn

6. Do you stand out? Are you compelling, engaging, useful,
unique, the best?
Do you provide what your potential customers want?
Straightforward for humans and searchbots to find your content?
© 2007-2012 Roland Dunn

7. Do you stand out? Are you compelling, engaging, useful,
unique, the best?
Do you provide what your potential customers want?
Straightforward for humans and searchbots to find your content?
© 2007-2012 Roland Dunn

8. © 2007-2012 Roland Dunn

9. “Googlebot ... may be unable to completely index
all the content on your site”
© 2007-2012 Roland Dunn

10. © 2007-2012 Roland Dunn

11. © 2007-2012 Roland Dunn

12. © 2007-2012 Roland Dunn

13. © 2007-2012 Roland Dunn

14. Doesn’t tell us what searchbots index
© 2007-2012 Roland Dunn

15. © 2007-2012 Roland Dunn

16. © 2007-2012 Roland Dunn

17. © 2007-2012 Roland Dunn

18. © 2007-2012 Roland Dunn

19. Webserver Logfiles ....
© 2007-2012 Roland Dunn

20. Webserver Logfiles ....
© 2007-2012 Roland Dunn

21. Webserver Logfiles ....
© 2007-2012 Roland Dunn

22. © 2007-2012 Roland Dunn

23. Over 2 months, 40% of all Googlebot’s requests
were to …. just 2 URL’s (on-site search URLs)
© 2007-2012 Roland Dunn

24. Over 2 months, 40% of all Googlebot’s requests
were to …. just 2 URL’s (on-site search URLs)
On a website with approximately 40,000 URLs in
Google’s index (allegedly, according to site:…)
© 2007-2012 Roland Dunn

25. Over 2 months, 40% of all Googlebot’s requests
were to …. just 2 URL’s (on-site search URLs)
On a website with approximately 40,000 URLs in
Google’s index (allegedly, according to site:…)
On a website serving approx. 150-200K unique
natural search visits/month
© 2007-2012 Roland Dunn

26. Over 2 months, 40% of all Googlebot’s requests
were to …. 2 URL’s.
On a website with approximately 40,000 URLs in
Google’s index.
On a website serving approx. 150-200K unique
natural search visits/month
© 2007-2012 Roland Dunn

27. Over 2 months, 40% of all Googlebot’s requests
were to …. 2 URL’s.
On a website with approximately 40,000 URLs in
Google’s index.
On a website serving approx. 150-200K unique
natural search visits/month
© 2007-2012 Roland Dunn

28. © 2007-2012 Roland Dunn

29. © 2007-2012 Roland Dunn

30. © 2007-2012 Roland Dunn

31. Categorise URLs by “top level section” – perhaps more
useful than just URLs ...
© 2007-2012 Roland Dunn

32. Table of “top level sections” requested by Googlebot
© 2007-2012 Roland Dunn

33. Chart of “top level sections” requested by Gbot – 6 months
© 2007-2012 Roland Dunn

34. © 2007-2012 Roland Dunn

35. Does Googlebot request all URLs served by a website?
© 2007-2012 Roland Dunn

36. Does Googlebot request all URLs served by a website?
No!
© 2007-2012 Roland Dunn

37. How Does Googlebot Spend Its Time? What Does it Request?
© 2007-2012 Roland Dunn

38. Does Googlebot spend its time cost-effectively?
© 2007-2012 Roland Dunn

39. Does Googlebot spend its time cost-effectively?
Not always.
© 2007-2012 Roland Dunn

40. © 2007-2012 Roland Dunn

41. Googlebot does not always request all content
© 2007-2012 Roland Dunn

42. Googlebot can become distracted, obsessed, or even lost:
© 2007-2012 Roland Dunn

43. Googlebot can become distracted, obsessed, or even lost:
• On-site search
• Additive filters/faceted URLs e.g.
“shoes?size=3&colour=green&price=10&brand=smith”
• Sections with thin and/or very similar content
© 2007-2012 Roland Dunn

44. Googlebot may not spend its time efficiently
It needs help focusing on what we value
© 2007-2012 Roland Dunn

45. © 2007-2012 Roland Dunn

46. SMX West, March 2011, Matt Cutts:
(http://goo.gl/ZZz7E):
“... if Google determines a site isn’t as useful to
users, they may not crawl it as frequently"
© 2007-2012 Roland Dunn

47. © 2007-2012 Roland Dunn

48. Suggestions:
• Embrace logfiles – full of useful information
© 2007-2012 Roland Dunn

49. Suggestions:
• Embrace logfiles – full of useful information
• Periodically check your searchbot behaviour
© 2007-2012 Roland Dunn

50. Suggestions:
• Embrace logfiles – full of useful information
• Periodically check your searchbot behaviour
• Especially if the “Googlebot found a high number” message
keeps appearing
© 2007-2012 Roland Dunn

51. If searchbots appear distracted, unfocused, lost:
© 2007-2012 Roland Dunn

52. If searchbots appear distracted, unfocused, lost:
• Alter internal navigation & linking (e.g. flatten hierarchy)
© 2007-2012 Roland Dunn

53. If searchbots appear distracted, unfocused, lost:
• Alter internal navigation & linking (e.g. flatten hierarchy)
• Robots.txt out (blunt approach)
© 2007-2012 Roland Dunn

54. If searchbots appear distracted, unfocused, lost:
• Alter internal navigation & linking (e.g. flatten hierarchy)
• Robots.txt out (blunt approach)
• Alter URL construction (expensive!)
© 2007-2012 Roland Dunn

55. If searchbots appear distracted, unfocused, lost:
• Alter internal navigation & linking (e.g. flatten hierarchy)
• Robots.txt out (blunt approach)
• Alter URL construction (expensive!)
• Check “sitemaps” – all URLs? What priorities expressed?
© 2007-2012 Roland Dunn

56. © 2007-2012 Roland Dunn

57. Lost Children ... or ... Hungry Psychopaths?
© 2007-2012 Roland Dunn

58. Or .... Distracted Teenager?
© 2007-2012 Roland Dunn

59. © 2007-2012 Roland Dunn

60. Sonification: What do Search Engines Requests Sound Like?
© 2007-2012 Roland Dunn

61. Sonification: What do Search Engines Requests Sound Like?
© 2007-2012 Roland Dunn

62. © 2007-2012 Roland Dunn

63. Online Branding: Natural Search:
http://www.refinedpractice.com/ http://www.cloudshapes.co.uk/
T: @RefinedPractice T: @roland_dunn
Slides Available At:
http://www.cloudshapes.co.uk/talks/
© 2007-2012 Roland Dunn