Crawling ecommerce sites – Maria Camanes' top tips from Tea-time SEO

1. Crawling large ecommerce sites

2. Maria Camanes, Senior SEO Consultant
● Over 6 years in SEO. Now a Senior SEO Consultant at Builtvisible,
where I joined 3 years ago
● Passionate about the technical side of SEO and specialised in site
speed optimisation and eCommerce SEO
● Work across a variety of accounts but mostly eCommerce sites
● Occasional speaker and regular trainer at BrightonSEO
● Twitter @mariacamanes

3. Common issues:
• A missing or wrongly implemented product retirement strategy can – and will – have a
negative impact on any ecommerce site’s organic performance
• Broken links are harmful for all types of sites but the possibility of broken links in an
ecommerce site is higher
• Discontinued or temporarily unavailable products can result in large quantities of 404s,
broken links and empty category pages (thin content)
• Displaying a 404 or empty page to your beloved customers will result in bad UX but also on
large quantities of link equity being lost
Today we’ll focus on how to find out of stock products as well as thin category pages and
- as these often occur in large quantities - how to deal with them at scale.
Maria’s tips on crawling large ecommerce sites

4. For example:
• This product page, has 91 backlinks from 28 different referring domains. The site has a
number of pages that return a 4xx status code with a significant number of backlinks
• As a result, it’s quite common to find large amounts of out of stock product pages for a
single site indexed by search engines
Tip #1:
Crawl your site to find out of stock products at scale

5. How to do it:
Use the ‘Custom search’ feature in Screaming Frog to find these when they return a 200 status
code (as these won’t be picked up via a standard crawl or GSC)
• Step 1: find the identifiable out of stock copy on your product page. Using Amazon as an example, their
identifiable out of stock is “currently unavailable”
Tip #1:
Crawl your site to find out of stock products at scale

6. How to do it:
Use the ‘Custom search’ feature in Screaming Frog to find these when they return a 200 status
code (as these won’t be picked up via a standard crawl or GSC)
• Step 1: find the identifiable out of stock copy on your product page. Using Amazon as an example, their
identifiable out of stock is “currently unavailable”
• Step 2: copy and paste this into Screaming Frog’s ‘Custom Search’ feature and run a crawl
Tip #1:
Crawl your site to find out of stock products at scale

7. How to do it:
Use the ‘Custom search’ feature in Screaming Frog to find these when they return a 200 status
code (as these won’t be picked up via a standard crawl or GSC)
• Step 1: find the identifiable out of stock copy on your product page. Using Amazon as an example, their
identifiable out of stock is “currently unavailable”
• Step 2: copy and paste this into Screaming Frog’s ‘Custom Search’ feature
• Step 3: crawl will return all of the product pages that contain the “out of stock” string. Don’t forget to manually
QA for any errors
Tip #1:
Crawl your site to find out of stock products at scale

8. • You can use the same process to find product listing pages that are empty (meaning they
have no products)
• Just copy the ‘no products’ identifier in Screaming Frog, in the same way we did for ‘out of
stock’ products
• Here are some examples:
Tip #2:
Apply this to category pages to find empty PLPs

9. Common issues:
• Thin category pages with limited stock are also a source of bad UX
• They will result in lost sales and when this happens at scale, this can have a significant
impact in revenue (not only for SEO)
• They put the site at risk of algorithm penalties
Tip #3:
Use the ‘Custom extraction’ tool to find thin PLPs

10. How to do it:
Taking this ASOS category page as an example
• Step 1: fetch the contents of any container using class=“styleCount” by typing this: //*[contains(@class,
'styleCount’)]
Tip #3:
Use the ‘Custom extraction’ tool to find thin PLPs

11. How to do it:
Taking this ASOS category page as an example
• Step 1: fetch the contents of any container using class=“styleCount” by typing this: //*[contains(@class,
'styleCount’)]
• Step 2: when crawl finishes, go to the ‘Custom’ tab at the top of the tool and you’ll see something like this
Tip #3:
Use the ‘Custom extraction’ tool to find thin PLPs

12. How to do it:
Taking this ASOS category page as an example
• Step 1: fetch the contents of any container using class=“styleCount” by typing this: //*[contains(@class,
'styleCount’)]
• Step 2: when crawl finishes, go to the ‘Custom’ tab at the top of the tool and you’ll see something like this
*Note: if your page doesn’t have a container with the number of products available, you can still count the number
of elements on a page: count(//div[@class="offer__content"])
Tip #3:
Use the ‘Custom extraction’ tool to find thin PLPs