The evolution of the web and the rise of the latest technologies has created a whole new level of complexity for SEOs, making it difficult to be able to crawl, analyse and understand their websites. If the more traditional elements of SEO such as internationalisation and faceted navigation weren't hard enough to get right already, we now have frameworks like JavaScript and configurations like AMP and PWAs to deal with as well. In this talk, Rachel will walk you through exactly how to analyse the full scale of your website and collect data on all of the pages it has available, even those served by advanced technologies. She will also provide best practice advice on how to optimise and configure your website to make sure only its most valuable pages are being served to users and search engines.

2. 3. Consolidating your pages for
better performance in search.
1. New web technologies that impact
SEO and how to handle them.
2. How to make your pages crawlable
and accessible.
WHAT WE’LL COVER

3. THE LATEST
CHALLENGES IN
TECHNICAL SEO
WHAT WE’RE UP AGAINST NOW

4. The latest challenge:
Keeping up with
developers.

5. The latest challenge:
Keeping up with
developers.
And helping Google
keep up too.

6. Developers keep
building
technologies that
we need to learn
how to maintain.

7. WebTechnologies
SearchEngines

8. SEOs
WebTechnologies
SearchEngines

9. Source: https://twitter.com/trypnotik/status/1121089330473160706

10. SEOs don’t know
enough about web
technologies, and
devs don’t know
enough about search.

11. This is a huge
problem.

12. JAVASCRIPT

13. JavaScript is everywhere,
and JS interest and usage
are still on the rise.

16. Source: https://httparchive.org/reports/state-of-javascript#bytesJs

17. “JavaScript allows you to add complex
animations, fetch customer data, use
open APIs to integrate with popular
technologies, and enhance user
experience. JavaScript will never
disappear.”
Emma Wedekind
UX Engineer at LogMeIn

18. JavaScript opens
up a world of
opportunities for
developers...

19. ...but what about
for SEOs?

20. ?
...and for search engines?

21. CRAWLING JAVASCRIPT
Challenge:
Most search engines can’t render JavaScript or
crawl links generated by JavaScript.

22. Challenge:
Most search engines can’t render JavaScript or
crawl links generated by JavaScript.
Solution:
Implement pre-rendering or server-side rendering
for improved link discovery.
CRAWLING JAVASCRIPT

23. Challenge:
Processing JavaScript is an expensive and
time-consuming process.
PROCESSING JAVASCRIPT

24. Solution:
Prioritise important resources for rendering and
optimise the critical rendering path.
Challenge:
Processing JavaScript is an expensive and
time-consuming process.
PROCESSING JAVASCRIPT

25. Challenge:
Testing JavaScript rendering can be difficult, but we
need to see how search engines are impacted.
TESTING JAVASCRIPT

26. Solution:
Use a headless browser and DOM comparison
tools to analyse contents of output code.
Challenge:
Testing JavaScript rendering can be difficult, but we
need to see how search engines are impacted.
TESTING JAVASCRIPT

27. Compare source code and rendered code.
Diffchecker

28. If your website relies on JavaScript
to power its content and
navigation, search engines could
end up seeing a blank screen with
nothing to crawl or index.

29. JavaScript
enabled
JavaScript
disabled

30. JavaScript disabled:
No body content, just a footer.

31. Crucial content should be
accessible without JavaScript
enabled and without the client
having to render.

32. Search engines are focused on generating
markup from templates and data.

33. Load event in
Chrome
DevTools.

34. Google takes a snapshot of the loaded
HTML at this point for indexing.

35. Crucial content
and scripts
should be
loaded before
the red line.

36. JavaScript can be crawled.
DeepCrawl

37. PWAs

39. Focuses on the core content of a
page, then progressively adds
more technically advanced
features on top if the conditions
of the client allow for this.
PROGRESSIVE ENHANCEMENT

43. Challenge:
PWA pages can generate content dynamically, but
search engines can’t interact like a user can.
CRAWLING PWAs

44. Solution:
Make sure key content can be accessed from a
unique URL and is served in the HTML.
Challenge:
PWA pages can generate content dynamically, but
search engines can’t interact like a user can.
CRAWLING PWAs

45. Challenge:
PWAs are often built using complex JavaScript
frameworks which can cause rendering issues.
RENDERING PWAs

46. Solution:
Perform thorough rendering testing with tools like
the Mobile-friendly Test and Chrome DevTools.
Challenge:
PWAs are often built using complex JavaScript
frameworks which can cause rendering issues.
RENDERING PWAs

47. PWAs are crawlable.
DeepCrawl

48. You don’t have to be an
expert in every platform
to be able to do an SEO
audit on them.
You just need to know
what to focus on.

49. All unique content needs to
exist on its own crawlable
URL.

50. Use descriptive titles for
each page to give context
on what page is about.

51. Use semantic HTML
markup for key page
elements.

52. Ensure crucial resources
aren’t being blocked in
robots.txt.

53. Implement pre-rendering or
server-side rendering so
content is accessible.

54. Make sure output code
contains the content you
want to rank with.

55. Use clean <a> links to
ensure link discoverability
and crawlability.

57. Checking the site isn’t broken,
leaving search engines with
nothing to crawl and index.
Improving the site’s chances of
ranking in search.

58. Your pages will now be crawlable
and accessible for search
engines, but there’s a final step.
Making sure they can perform in
search.

59. CONSOLIDATING
YOUR WEBSITE’S
SIGNALS
IMPROVING PERFORMANCE

60. We’ve bridged the gap, but search
engines still need additional help.

61. When it comes to
setting preferred URLs,
Google often
assumes you are
wrong.

62. Canonical URL:
example.com/product
Internally linked URL:
example.com/product?facet= ...
Externally linked URL:
example.com/product?utm= ...
Sitemap URL:
example.com/product-page
?

63. Googlebot is confused,
and will assume the site
owner needs some help.
So it will pick a canonical
URL for you.

64. An algorithm will be used
to help make sense of this
mess and determine the
canonical.
It uses these signals:

65. Parameter
Handling
Canonicals
Internal &
External Linking
Hreflang Links
Redirects
Sitemaps

66. DIFFERENT SIGNALS DETERMINE GOOGLE’S
CANONICAL SELECTION
Rel canonical, redirects, internal
linking, URL parameters and
sitemaps are all signals Google
uses to decide which page is the
canonical from a group of pages
that have been folded together.
John Mueller, Google Webmaster Trends Analyst
Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-june-26th-2018/

67. If you sit back and
do nothing, Google
will do its best to
pick preferred URLs
for you.

68. But the algorithm
isn’t perfect and it
doesn’t know your
site as well as you.

69. The algorithm’s
results will be
better if you use
website signals
consistently.

70. Check for any
inconsistencies in
the signals you’re
sending to search
engines.

71. 1. Internal linking
2. Redirects
3. Noindex
4. Sitemap inclusion
5. Disallow
@rachellcostello SearchLeeds

72. Links in to canonicalized vs canonical pages.
Redirecting canonical pages.
Non-200 canonical pages.
Inclusion of canonical pages in sitemaps.
Non-indexable pages being shown in the SERPs.
Some areas to investigate:

73. The best way to see
Google’s URL selection for
yourself is through the
cache and Google Search
Console.

74. Go to Google to see the results...

75. Incorrect cached page suggests a
different canonical has been chosen.

76. YOU CAN SEE WHICH CANONICAL GOOGLE
HAS CHOSEN IN THE NEW SEARCH CONSOLE
In the new Search Console you
can select individual URLs which
takes you to the info query,
showing you the canonical that
Google has chosen.
John Mueller, Google Webmaster Trends Analyst
Source: https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-january-9th-2018/

77. Let’s take a look.

78. The key reports in Google Search Console
for canonicalisation analysis...

80. “This page is marked as canonical, but
Google thinks another URL makes a better
canonical which has been indexed instead.”
Source: https://support.google.com/webmasters/answer/7440203?authuser=0#google_chose_different_canonical_than_user

82. “The URL is one of a set of duplicates
without an explicitly marked canonical
page. Google did not index this URL.
Instead, we indexed the canonical that we
selected.”
Source: https://support.google.com/webmasters/answer/7440203?authuser=0#submitted_url_not_selected_as_canonical

83. With website
signals, there’s
strength in
numbers.

84. Make website signals
clear and consistent for
Googlebot so it doesn’t
need to make decisions
on your behalf.

85. Checking the site isn’t broken,
leaving search engines with
nothing to crawl and index.
Improving the site’s chances of
ranking in search.

86. 3. Use consistent signals to ensure
your pages perform in search.
1. Keep up to speed on the latest
technologies devs are building.
2. Make sure key content and links
are accessible for search engines.
KEY TAKEAWAYS #SMXInsights

87. https://marketing.deepcrawl.com/acton/media/31628/javascript-seo-white-paper
https://www.youtube.com/watch?v=wSwzfEn5-6A&list=PLKoqnv2vTMUPOalM1zuWDP9OQl851WMM9
https://builtvisible.com/progressive-web-app-seo/
https://www.deepcrawl.com/blog/news/google-webmaster-hangout-notes-june-26th-2018/
ADDITIONAL RESOURCES