Imagine having a big brand website with over 50% of its content not indexed by Google after weeks? Now imagine that the issue causing this severe decline in online visibility was self-induced? In the new digital marketing landscape, big brands losing money this way is shockingly normal. This deck from a Searchmetrics webinar Bartosz Goralewicz did with Marcus Tober on December 4, 2019. Want to watch the video? https://youtu.be/3XENVxVpyoo

1. Bartosz Góralewicz
linkedin.com/in/goralewicz @bart_goralewicz
Indexing Your
Content is Optional
2020 Edition

2. 63% of Walmart’s pages are not indexed in Google.
H&M is losing 4,938,885 visits per month due to their indexing issues.
24% of Topshop products are not indexed in Google.
Now that I have your full attention – let’s begin J

3. Back in 2016 I published Hulu’s case study…
… that I wrote in SearchMetrics’ office in Berlin J
one.ly/hulu2016

4. In 2016 JavaScript
was… exotic?

5. Hulu vs. Netflix – the first JavaScript SEO battle

7. Fast forward to 2019

8. HTML HTMLHTML
There is no such thing
as an HTML website!

9. In 2019
* This means that their content isn’t
visible with JavaScript switched off.
** this also means that Google
needs to render the page to see
that content.
Rendering is optional though.
of the websites use
JavaScript for their
crucial content.*80%

10. Brands with content relying
on JavaScript

11. 82%
0%
0%
42%
73%
0%
22%
Percentage
of
JavaScript
content
indexed

12. Let’s have a look
at the indexing issue
in the wild

13. case study

15. Let’s compare H&M’s crawl
with and without JS rendering

16. Let’s compare H&M’s crawl
with and without JS rendering

17. Read more here:
one.ly/hm

18. What usually relies on JavaScript?
01
Pagination
02
“You might be also
interested in” (internal linking)
03
Top products
04
Reviews
05
Comments
06
Main content (rarely)

19. The Disconnect
Google’s announcements this year didn’t help…
Statement:
Martin and John are nothing but helpful.
Google is a large organization though
and sometimes they make our life a little
bit too complex.

20. But… but 90% of the websites get
indexed in 5 seconds?

21. one.ly/martin5s

22. What is the exact timeframe for
JavaScript indexing?
Unfortunately - NO.
5 seconds
for 90% of
the pages?

23. Yeaaaah, my JS content can get indexed in 5 seconds.
Well, not exactly.

28. Today’s episode was sponsored by the phrase:
Queue latency

29. What Martin actually meant

30. Ranking

31. There is no timeframe for
JavaScript indexing!

32. JS indexing delays are still an issue
Good Bad

33. It is 2019 and our
research shows that
there are hundreds
of thousands of
domains not fully
indexed.
Even after… Months of
publishing the content

34. Let’s get technical…

35. Web development evolved. Most websites use ready-to-go components
Slider
Faceted navigation
Related Products
Comments
Main content
Menu
The roots of the problem

36. Slider
Faceted navigation
Related Products
Comments
Main content
Menu
So when Google isn’t rendering your website,
it may see…well, this:

37. 2 completely different websites
and 2 different website graphs.
Rendering delays lead to

38. Category Ladies/Men/DividedCategory Category
H&M.comHOMEPAGE
Sub-Category Sub-Category Sub-Category Jeans
Products Products Products Products Shaping Skinny High Jeans
JavaScript

39. Category Ladies/Men/DividedCategory Category
H&M.comHOMEPAGE
Sub-Category Sub-Category Sub-Category Jeans
Products Products Products Products Shaping Skinny High Jeans
JavaScript

40. Let’s compare H&M’s crawl
with and without JS rendering

41. Let’s compare H&M’s crawl
with and without JS rendering

42. Let’s talk rendering…

43. Rendering is not Google’s
indexer default mode.
Rendering is triggered by... Heuristics

44. Rendering heuristics
So, you look at the difference
between the initial HTML, and,
then, if after rendering you
see extra content?
Yeah.

45. There are certain heuristics, that,
if we see after a while, like, oh, this
page, actually, the renderer does not
diff as much or doesn’t diff, it looks
the way before
The Heuristics
I still haven’t fully, like, grasped
what exactly triggers the
heuristics

46. are FAR from
perfect though.
…those heuristics

47. Every website, when we see
them for the first time, goes
to rendering.
ALL NEW SITES
GET RENDERED

48. But what is
a new website exactly?
What is a new website?

49. Rendering Heuristics
Logic behind
those heuristics
Limitations Changes in how we should
approach web development
• Staging
• Experiments

50. Rendering vs. Crawler budget

51. Rendering vs. CPU cost

53. WRONG

55. CPU is not
affecting
rendering*
*within what we would call
a “reasonable” load

58. Is the JavaScript rendering cost
affecting crawling and indexing?
It’s complicated

59. How to check the rendering
cost of a page
TL;DR – Too Long Didn’t Render
onely.com/tools

60. TL;DR – Too Long; Didn’t Render
onely.com/tools

61. onely.com/tools

62. onely.com/tools

63. Content indexing
- timeframe

64. Percentage of
JavaScript
content
indexed
after 14 days
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
100%
96%
70%
34%
0,5%
New York Post Eventbrite Target The Guardian CNBC

65. TGIF - The Google Indexing Forecast
onely.com/tools/tgif

66. HTML vs. JavaScript indexing

67. HTML vs. JavaScript indexing

68. Diagnosing
rendering issues

69. WWJD - What Would JavaScript Do

70. WWJD – BBC & meta tags vs. rendering

71. WWJD – BBC & meta tags vs. rendering

72. WWJD – BBC & meta tags vs. rendering

73. WWJD – BBC & meta tags vs. rendering

74. WWJD – Links added by JS

75. WWJD – Links removed by JS

76. JSLet's talk about
HTML

77. Even without JavaScript dependencies
big players struggle with indexing
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Indexed after 1 day Indexed after 2 days Indexed after 1 week Indexed after 2 weeks
The Guardian Target Reuters Eventbrite
98.5%
3.5%
68.5%
32%
99.9%
33%
88%
47%
99.9%
78%
91%
57%
99.9%
84%
91%
58%

78. Wrapping up

79. Parts of
your website
depend
on JS
01
Webmaster updates
with new products
02
Google crawls your
website without
seeing all the links
03
Only crawls a part of the
domain without finding
products (valuable content)
05
Budget is too
low to render JS
04
Google indexer is
confused - crawler
budget falls

80. Which leads to this…
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Indexed after 1 day Indexed after 2 days Indexed after 1 week Indexed after 2 weeks
The Guardian Target Reuters Eventbrite
98.5%
3.5%
68.5%
32%
99.9%
33%
88%
47%
99.9%
78%
91%
57%
99.9%
84%
91%
58%

81. TO DO:
Go to onely.com/tools
and check your website
(100% free)
Crawl your website with
and without JS and
compare two crawls

82. We are going like: ”Oh, all right, we
are gonna skip rendering.” It is not
as frequently happening anymore.
(...) Many websites even if they do
not run JavaScript, they might still
go through the render phase,
because it doesn’t make a difference
as much
Crawling
and indexing
to come
together
I expect, eventually rendering,
crawling and indexing will come
closer together

83. Summary
Indexing your content
is one of the key
challenges for 2020
All indexing issues
we are seeing are
100% self-induced by
website owners
We need to redefine
how we are looking at
the crawler budget and
the factors affecting it
JavaScript SEO
= Technical SEO

84. To do:
1Closely monitor your
indexing stats (daily,
weekly, biweekly).
2
If your content relies on
JavaScript, monitor it closely
and make sure that:
• The code is light and efficient
(cheap to render)
• Your code doesn’t affect web
performance on low-end devices
3
Make sure that no
content within your
structure relies on
JavaScript (check all
the page types within
your domain).
4 Make sure that no meta data
is changed during rendering.

85. To do:
1Closely monitor your
indexing stats (daily,
weekly, biweekly).
2
If your content relies on
JavaScript, monitor it closely
and make sure that:
• The code is light and efficient
(cheap to render)
• Your code doesn’t affect web
performance on low-end devices
3
Make sure that no
content within your
structure relies on
JavaScript (check all
the page types within
your domain).
4 Make sure that no meta data
is changed during rendering.
Subscribe to Onely.com – be the
first one to know about our
crawl budget monitoring toolset.

86. More data is coming soon

87. Cloaking in 2019
bartosz@onely.com
linkedin.com/in/goralewicz @bart_goralewicz
THANK YOU
one.ly/15min