Crawl before you run... with enterprise SEO Enterprise websites are often large, sprawling things with dark corners people only find out about years later. In this webinar with Amanda King, SEO Specialist at Optus, we’ll go through why it’s important you understand all the dark corners of your website (and how it impacts Google and other search engines seeing the important bits of your site). Plus, if you’re so inclined, we’ll go through some easy ways to find potential rabbit holes, and other analysis and review to take to your teams or agencies.

1. Crawl Before You Run
Amanda King & In Marketing We Trust
19 Nov 2020

2. 2
What’s on
What is crawling &
indexation
1
Why does it matter?
2
What affects website
crawling & indeation
3
What are some
common issues
4
Where does my
website sit currently
5
How can I improve my
crawling & indexation
6
About me
7

3. 3
What is crawling &
indexation?

4. 4
Basically, Google follows links on
your website (and others) to see the
Internet. All the time.
Source: https://www.google.com/search/howsearchworks/crawling-indexing/

5. 5
...but it’s got a limited attention span,
because crawling the entire Internet
is tiring and expensive.
Source: https://webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-googlebot.html

6. 6
So, Google won’t always crawl
(read) your entire website, and even
if it does, it won’t always index
(remember) your entire website.
Source: https://webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-googlebot.html

7. 7
So, Google won’t always crawl your entire website, and even if it does, it won’t
always index your entire website.
Source: https://webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-googlebot.html
Source: https://searchengineland.com/figz/wp-content/seloads/2020/09/image7.jpg

8. 8
Google is like a toddler on
a sugar high.
If you make it’s life easy,
it’ll keep running around
finding things, but if you
make it challenging, it hits
a wall and may not come
back, at least for a while.
Source: https://tenor.com/view/sugar-kick-high-wild-kid-gif-5967980
https://gfycat.com/discover/sleepy-toddler-gifs

9. 9
So what? Google
only needs to see my
website once, right?

10. 10
Nope!

11. 11
Just as your website isn’t the same
as it was the day it was published,
Google changes how it understands
the content and relevance of your
website. Constantly.

12. 12
This means the most important
pages of your website may be
crawled and indexed daily.
https://www.searchenginejournal.com/google-crawl-indexing/274141/#close

13. 13
If you consistently make it difficult for
Google to see your website,
eventually they may give up.
And that could have massive
ramifications for your business.
https://www.searchenginejournal.com/google-crawl-indexing/274141/#close

14. 14
Google needs to see your website in
order to understand it, and serve it
for searches. Why?
https://support.google.com/webmasters/answer/70897?hl=en

15. 15
Remember Google’s mission: to
organise the world’s information

16. 16
Because that’s the way it’s built.
Two major factors in how
frequently or infrequently
your website is crawled
(and ranked!) are:
- popularity
- staleness

17. 17
Because that’s the way it’s built.
Over the years that’s been
considered in algorithm
and serving updates like:
- QDF (Query Deserves
Freshness)
- BERT / RankBrain / AI
/ ML
- Expertise - Authority -
Trust (EAT)
Source: https://media0.giphy.com/media/26BRvrzQUM9EYc8Ks/source.gif

18. 18
So what else do I
need to know?

19. 19
So what else do I need to know?
There is an ideal website
structure for Google
Google crawls, indexes
and ranks the mobile
version of your site
Source: https://www.impactplus.com/blog/best-site-structure-google

20. 20
So what else do I need to know?
https://whatdoesmysitecost.com/
Javascript makes things
more difficult in a few ways
- It requires Google to
crawl your page twice
- It (often) increases the
weight of your page,
which increase the
time it takes Google to
read one of your
pages
Source: https://developers.google.com/search/docs/guides/javascript-seo-basics; https://media.tenor.com/images/fd70dbcd2ae4b786d9f7fd75f2efa870/tenor.gif

21. 21
What are some
common problems?

22. 22
Do you have an
ecommerce website where
you can filter by everything
under the sun…
...and it creates it’s own
category page?
Like, someone wanted to
find a blue gaming laptop
under $1,000 and that
created it’s own page?
https://media1.tenor.com/images/251da9c5c43d4bf2e4ddb36ed6d4c2eb/tenor.gif?itemid=10603325
Faceted Navigation

23. 23
Faceted Navigation
Source: https://moz.com/blog/large-site-seo-basics-faceted-navigation

24. 24
Keep pages out of the
index that your customers
don’t need to (or shouldn’t)
be seeing.
- Thank you pages
- Confirmation pages
- Internal site search
results
Source: https://media.tenor.com/images/37bb70acdf42cb0f61e599a16c2dd606/tenor.gif

25. 25
Sometimes we tell Google
conflicting or unclear things
in page level commands in
places like:
- robots.txt
- meta robots
- x-robots
- canonicalisation
Page level command issues
Source: https://media1.giphy.com/media/8gRgYeXD3rKUcjWlzB/giphy.gif; https://www.seroundtable.com/noindex-canonical-google-18274.html

26. 26
Sometimes, Google will
just skip reading parts of
your webpage that aren’t
directly related to the main
topic. This could be
sections like:
- “Related Topics”
- “Similar products”
Layout-based partial indexing
Source: https://searchengineland.com/how-to-earn-your-place-in-googles-index-in-2020-340498;
https://i.pinimg.com/originals/e6/90/16/e69016cb58ee9127d967bc09f7af8c74.gif

27. 27
So how do I know
how I’m doing?

28. 28
Google Search Console
should tell you what
Google is crawling, and a
site: search will give you
an idea of what they’re
indexing.
If they’re wildly different,
something’s up.

29. 29
And what can I do to
improve it?

30. 30
- Site architecture
improvements OR
internal linking
- Fix any Javascript or
responsive design issues
- Make sure your sitemap
is up to date
Source: https://i.gifer.com/Dzq1.gif

31. 31
And who am I to tell
you this?

32. 32

33. 33
Amanda King
SEO Specialist
Optus
Email: amanda.king@optus.com.au
Contact details

34. Thank you