[MozCon 2019] How do you turn an unwieldy 2.5 million-URL website into a manageable and indexable site of just 20,000 pages? Areej will share the methodology and takeaways used to restructure a job aggregator site which, like many large websites, had huge problems with indexability and the rules used to direct robot crawl. This talk will tackle tough crawling and indexing issues, diving into the case study with flow charts to explain the full approach and how to implement it.

1. Fixing the Indexability Challenge:
A Data-Based Framework
Areej AbuAli
#MozCon 2019
slideshare.net/areejabuali@areej_abuali

2. I’m here to talk to you about a framework
that I came up with to fix a client’s
indexability challenge

3. But more importantly…
(And I only realised this after I’d finished preparing this talk)

4. I’m also here to talk to you about how
“Technical Problems are People Problems”

5. I first met this client back in 2017

6. They’re a job aggregator site

7. Their website was struggling

8. 89%YoY decrease in organic visibility
0
2000
4000
6000
8000
10000
12000

9. Organic Traffic
Y1 X
Y2 Y
Change -46%

10. They barely ranked…

11. 0
100,000
200,000
300,000
400,000
500,000
600,000
Competitor OrganicVisibility
Client indeed.co.uk monster.co.uk reed.co.uk

12. And their site was so massive;
we struggled to crawl it

13. To make this work, we need to fix the
fundamentals first

14. We need to fix their tech

15. So this talk is about my 18-month
relationship with this client

16. It’s about what worked, what didn’t work
and what I would have done differently

17. The Initial Findings

18. Links Tech Content
I started working on a comprehensive audit

19. I ended up with a 70-page document

20. There was a total of 50 recommendations

21. Some of the main things included were…

22. 72% of backlinks came from only
3 referring domains

23. Their on-page content was full of
duplication and missing the basics

24. There were NO sitemaps

25. Canonical tags were not set up correctly

26. And their internal linking structure
was a nightmare

27. Every recommendation was outlined using a
traffic light system

28. And every section was split into
Problem, Effect & Solution

29. We had a half-day audit handover meeting
where I walked them through all
of our recommendations

30. Everyone was in good spirits

31. Yet I couldn’t help but feel it
was not enough…

32. Something was missing

33. Everything I recommended up till
now was solid

34. But I had a gut feeling that due to the
nature of their site…

35. These recommendations wouldn’t
quite cut it

36. I had to go back to the drawing board

37. The Supplementary Findings

38. Also Known As…

39. The Findings I Should’ve Found The First Time
Round But Didn’t So I’m Choosing To Call It
Supplementary Findings To Sound
Like An Expert.

40. They’re a job aggregator site – in essence, they’re a
job search engine

41. That means that every single search
conducted could, potentially, be
crawled and/or indexed
if it wasn’t built right

42. That’s equivalent to an infinite number
of potential searches!

43. Which part was their site
not getting right?

44. I knew that it was impossible to crawl

45. The one time I tried to fully crawl the site,
it returned over 2.5 million URLs

46. And I could only crawl it by excluding
massive sections

48. And if their pages couldn’t be crawled,
then they would never be
indexed properly…

49. And they won’t rank

50. So there were three problems that
needed to be fixed

51. Crawling Indexing Ranking

52. It was apparent that there were no rules in
place to help direct robots

53. Unique URLs were created, using up all
possible filter combinations

54. This can create a potentially-unlimited
number of URLs

55. And all the pages looked exactly the
same – they just had a list of jobs!

56. Google was wasting crawl budget by
crawling duplicate thin pages and
attempting to index them

57. The ‘Aha!’ Moment

58. It was apparent that there were no rules in
place to help direct robots

59. I needed to create a customised framework
that instructs bots on what to do and
what not to do

60. The job aggregator industry seemed to be
doing one of two things

61. Limiting indexable pages
à miss out on ranking opportunity

62. Limiting indexable pages
à miss out on ranking opportunity
Not limiting indexable pages
à wind up with reduced link equity

63. My framework was going to do neither

64. Instead it would use search volume data
to determine which pages are
valuable to search engines

65. So how exactly would this work?

66. It starts off by passing
the search query
through a keyword
combination script

67. This script outputs different combinations
for the search conducted

68. It does that by changing the order of
keywords to see all possible combinations

69. Digital Marketing Manager London

70. Digital Marketing Manager London
London Digital Marketing Manager
Digital Marketing London Manager

71. 4! = 24 combinations

72. These combinations will increase based
on the search query and filters applied

73. Even though Google will regard most
of these combinations as the same…

74. The script will help avoid duplicating
pages that have different versions
of the same thing

75. It then searches the
database to see
if this job is available

76. If (Job # Available)
Load a page stating so
and no-index it

77. If (Job = Available)
Search the keyword
database for all
keyword combinations
from the script

78. Fetch search volume
data for these
keyword combinations

79. For search volume data, we recommended
using keywordtool.io API

80. If (SearchVolume > 50)
A page is then created
using the highest SV
keyword combination
that is both indexed
and crawled

81. If (SearchVolume < 50)
Load page for users
but no-index it

82. Your search volume cut-off can be updated
at anytime and is based on what makes
sense for your industry

84. There is always the possibility of
errors occurring

85. What if there’s a tie-breaker?
(Several keyword combinations having the
same search volume)

86. Create an indexed crawlable page based
on the keyword used in the
user search query

87. What if the API was down?
(And you’re unable to generate search volume data)

88. Load a no-index page for usability
and don’t store the query in the database

89. As for their internal linking structure…

90. This was the status of their header

91. This is what we recommended

92. We also provided internal linking
recommendations for their footer and
job advertisement pages

93. And an exact breakdown of their filter
system on job search pages

94. As for their sitemaps – they
had none

95. We recommended creating and
splitting them up

96. Blog Sitemap
Job Advertisements Sitemap
Job Results Sitemap
Ancillary Sitemap

97. The final step was to help them sort out
their content

98. Even if they were only indexing high
search volume pages; their
content was very thin

99. Their chances to rank would
still be minimal

100. They had the same H1,Title Tag & Meta
Description for every filtered
indexed page

101. Most competitors automatically generate
optimised meta tags

102. They needed to do the same for
indexable filter pages

103. Other than a company page and a
handful of blog posts, there were
no core content pages

104. So we performed in-depth keyword research
and opportunity analysis to see what
content generates traffic

105. And we provided a content audit
and strategy to go with it

106. My ‘Aha!’ Moment felt complete

107. This was the piece of the puzzle
that was missing

108. Four Months Later

109. The client confirm that they
implemented everything

111. The first thing that caught my attention
was their site went from over
2.5M crawled pages to 20K

112. Which initially felt like good news…

113. Until I realised that their traffic had
declined…

114. Remember this?
Organic Traffic
Y1 X
Y2 Y
Change -46%

115. Organic Traffic
Y1 X
Y2 Y
Y3 Z
Change -86%

117. The Mini Audit Findings

118. Also Known As…

119. The Findings I’m Rushing To Find In A Panic
To Prove That They Haven’t Implemented
My Recommendations Accurately,
Hence I Am Still An Expert.

120. I went through the original list of
50 recommended actions

121. At that point, there were 29 that had
not been implemented

122. I also discovered ten new issues that were
affecting their indexability

123. Google was choosing to only index
20% of the submitted sitemap

124. Googlebot will choose to visit their site less
if their indexability is not in check

125. The client explained that they’ve added
canonical tags and felt that was enough

126. They’re relying on Googlebot to:
1) Crawl the pages

127. They’re relying on Googlebot to:
1) Crawl the pages
2) Find the canonical tags

128. They’re relying on Googlebot to:
1) Crawl the pages
2) Find the canonical tags
3) Then choose to ignore them

129. Canonical tags are simply hints for bots

130. Google might decide to ignore the
tags and pick other pages to index

131. Almost 80K pages had been indexed
despite not being submitted
via the sitemap

132. And some pages were included in the
sitemap but not indexed

133. /job/finance-manager-liverpool

134. /job/finance-manager-liverpool?index=
57271b7c~4&utm_campaign=job-
detail&utm_source=search-
result&utm_medium=website

135. Because this page was the one
discovered via internal links

136. Over 5K similar pages with parameters
were getting indexed

137. Your main goal is to maximise crawl budget

138. You cannot use canonical tags as a
sticking plaster to fix that

139. This implementation was still incomplete

140. I had to change my way of conveying
this message

141. I put a stop to the endless stream of emails
and scheduled a face-to-face meeting

142. We reviewed each and every single
remaining task and discussed them in detail

143. Behold the wonders of Google Sheets!

144. We re-prioritised tasks and put
estimated completion dates

145. It was not an easy meeting but
it felt productive

146. Where are we now?

147. On a personal level, I discovered that I
suffer from Imposter Syndrome

148. This was my constant state of mind

149. I was working closely with the CTO
throughout this project

150. I felt he didn’t trust me or my knowledge

151. So, what would I have done differently?

152. If I could go back in time, I would realise
what the actual problem was

153. All technical problems are people problems

154. The SEO recommendations were solid

155. Getting it implemented was the hard part

156. As a tech SEO, the most you can do
is to influence priorities

157. You have no control

158. In this instance, I didn’t manage to
persuade him to implement
the recommendations

159. I also learned that the way I’ve been
doing SEO audits is plain wrong

160. I always focused on delivering a set
of comprehensive actions

161. Instead, maybe I should just deliver
a SINGLE recommendation

162. And once that’s implemented…

163. Then, and only then, will I recommend
another

164. And maybe I shouldn’t recommend
Nice-To-Do’s…

165. Until there are only Nice-To-Do’s
left to do

166. Because they are simply a distraction from
the main problem

167. This talk does not have a happy ending…

168. It is not a successful case study…

169. There’s no upward visibility graph and
page one rankings for me to show off…

170. This talk is about real life

171. It’s about a framework I created that
fixes indexability issues that I’m proud of
and I know in my gut *works*

172. So I wanted to share it with you

173. Because I can see this applied across
many sites in plenty of industries

174. And I’d love forYOU to implement it

175. So I’m going to share my full
methodology with you

176. slideshare.net/areejabuali

177. bit.ly/mozcon-areej

178. And just remember…

179. Getting the basics right is so fundamental

180. If you can do nothing else,
just do the tech.

181. Areej AbuAli
Slides: slideshare.net/areejabuali
Tweet things @areej_abuali

182. References
(1) http://3.bp.blogspot.com/_zKFo7OwbIp8/TLkK8i5FrlI/AAAAAAAAEeY/oeXBnenxnWU/s1600/uphill3_l.jpg
(2) https://static.independent.co.uk/s3fs-public/thumbnails/image/2018/08/16/16/handshake.jpg
(3) http://www.subversivecopyeditor.com/.a/6a010536d5bba5970c022ad359d336200c-pi
(4) https://pbs.twimg.com/tweet_video_thumb/Dd4TQ7jUQAA-umE.jpg
(5) https://cdn-images-1.medium.com/max/1600/0*ZQDtKU0r347Sta1J.jpg
(6) https://b.kisscc0.com/20180923/xzw/kisscc0-three-wise-monkeys-cartoon-black-and-white-animal-three-wise-monkeys-simple-
5ba75b8cdb7c39.781586971537694604899.png
(7) https://upload.wikimedia.org/wikipedia/commons/c/ce/Puzzle_black-white_missing.jpg
(8) http://d70c4c94fa2afb6dbbc0-2bb54928637cb07488eb9dfab3a7ca9e.r6.cf2.rackcdn.com/uploaded/t/0e5859330_1485453015_too-good-to-
be-true-rotator.jpg
(9) https://imgc.allpostersimages.com/img/print/posters/kim-warp-an-aerial-view-of-a-car-driving-along-an-infinity-shaped-road-new-yorker-
cartoon_a-G-9164141-8419447.jpg