In this presentation, given at SMX Munich 2016, Joost talks about the methods he uses to do SEO for large scale websites. He covers his SEO data strategy, optimization strategy and regular audits.

1. Method to the madness - Large Scale SEO

3. What’s the problem?

4. Homepage
example.com/
Support portal
kb.example.com
Shop
example.com/shop/
News section
example.com/news/
A “modern” website:
Community portal
example.com/forums/

5. What could go wrong?
• Support portal outranking the main website
• An error in the shop software causing the entire site to
stop being indexed
• So much crawl budget being spent on community
forums that shop doesn’t get crawled properly
• KB linking to shop everywhere, shop changes URL
structure
• etc. etc. etc.

7. What’s the solution?

8. Modern websites cause a
need for a different SEO
approach.

9. Method
• Build properly
• Get all the needed data
• Optimize constantly
• Audit regularly
• Teach others

10. Do you know what’s on
your site?

13. Homepage
example.com/
Support portal
kb.example.com
Shop
example.com/shop/
News section
example.com/news/
It can be as simple as this:
Community portal
example.com/forums/

15. Method
• Build properly
• Get all the needed data
• Optimize constantly
• Audit regularly
• Teach others

16. A comprehensive SEO data strategy
• Service Oriented Approach to main sources of
information:
• Analytics
• Logs, specifically:
• Spider logs
• Error logs
• Internal Search

17. Analytics

18. Analytics
• Often the only thing that’s already centralized.
• Far too often answers questions that you don’t need to
answer.

19. Do you know what
content on your site is
the most engaging?

20. Facebook stats?

21. Twitter stats?

22. Pinterest?

23. Video?

24. This can be done.

25. Engagement metrics

26. Engagement metrics

27. Good analytics is an
aggegration of loads of
sources of data.

28. Logs & Search console

29. Googlebot stats before GSC tells you
Do you need Google Search Console to
tell you you’ve got errors on your site?

30. Logs
Centralize all logs into one “stash”.
My personal preference: Logstash on ElasticSearsch.
Using Kibana, you can filter out and create specific
dashboards for bot traffic and errors.

31. Googlebot stats that HELP

32. Some of my favorite views
Top spidered URLs over the last 24 hours, by Google
user agent.
Average response time, slow loading pages, slow
loading page types.
Googlebot 4xx and 5xx errors over the last 24 hours.

33. Google Search Console errors
Be honest, how many of you have “accepted” a “background
noise” of thousands or even 100k+ errors?

35. User errors
Is anybody at your company monitoring user “crashes”?
Pages that 404 more than X times a day?

36. If engagement is
king, errors are the
hangman.

37. Internal Search
Internal search

38. Internal search should
tell you which content
you have.

39. Internal Search monitoring
• Is someone / something monitoring the missed
keyword opportunities across your site?
• This could flow back into ElasticSearch / Kibana too

40. URLs

41. Homepage
example.com/
Support portal
kb.example.com
Shop
example.com/shop/
News section
example.com/news/
A link changes in your shop
Community portal
example.com/forums/

42. Homepage
example.com/
Support portal
kb.example.com
Shop
example.com/shop/
News section
example.com/news/
What happens to the links pointing there?
Community portal
example.com/forums/

43. URLs and links across your site
• In an ideal world, you never embed complete links.
• You embed ID’s, which are translated to links based on
one datasource.

44. Homepage
example.com/
Support portal
kb.example.com
Shop
example.com/shop/
News section
example.com/news/
One database that has all the URLs for:
Community portal
example.com/forums/

45. All URLs
• App deeplinks
• hreflang relations
• AMP URLs
• etc. etc.

46. XML Sitemaps & hreflang

47. Homepage
example.com/
Support portal
kb.example.com
Shop
example.com/shop/
News section
example.com/news/
You probably don’t have this:
Community portal
example.com/forums/

48. Homepage
example.com/
Support portal
kb.example.com
Shop
example.com/shop/
News section
example.com/news/
Community portal
example.com/forums/
Homepage
example.com/
Support portal
kb.example.com
Shop
example.com/shop/
News section
example.com/news/
Community portal
example.com/forums/
Homepage
example.com/
Support portal
kb.example.com
Shop
example.com/shop/
News section
example.com/news/
Community portal
example.com/forums/
Homepage
example.com/
Support portal
kb.example.com
Shop
example.com/shop/
News section
example.com/news/
But this, in 4, 5, 8, 20 languages:
Community portal
example.com/forums/

49. Multiple Sitemap indexes
• Don’t try to combine them!

50. Need to build XML sitemaps?
• Base it off of your Internal Search API!
• Use a smart naming scheme to help you dive into
sections of your site easily.

51. Method
• Build properly
• Get all the needed data
• Optimize constantly
• Audit regularly
• Teach others

52. Use your internal search
API for related content.

53. Feed engagement and
commercial metrics in.

54. Method
• Build properly
• Get all the needed data
• Optimize constantly
• Audit regularly
• Teach others

55. Site changes? Embed SEO into CI
• Who knows what CI is?

56. Create CI tests for important stuff
• Check canonicals
• Check hreflang
• Check that URLs stay the same
• Check HTTP header codes
• etc.

57. Regular crawls to identify issues
• Screaming Frog, OnPage.org, DeepCrawl, etc.
• I don’t care which one you use, as long as you one.

58. Who am I kidding?
Use OnPage.org ;)

59. Method
• Build properly
• Get all the needed data
• Optimize constantly
• Audit regularly
• Teach others

60. Teach developers
technical SEO.

61. Teach analytics people
to look beyond your
main analytics package.

62. Teach!

63. Questions?
Follow along:
• https://yoast.com/
• @jdevalk & @yoast on Twitter
• facebook.com/yoast