Acquire an All Access Pass to Club Google
•Als PPTX, PDF herunterladen•
1 gefällt mir•381 views
Optimising crawl budget and encouraging search engine indexing are concepts most SEOs are familiar with. But the devil is in the details. Especially as best practices have altered in recent years and will do so again with the introduction of APIs by both Google and Bing. Should you control crawlers with robots directives? Or XML sitemaps? Or submit via the APIs? Or just let Google figure it out? Let's into the optimal way to get your content into search engines fast.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Empfohlen
Empfohlen (20)
Acquire an All Access Pass to Club Google
Hinweis der Redaktion
- Have you heard? They opened a new nightclub. It’s so exclusive people literally wait for days in line, but most end up being turned away by the bouncer. But no matter when you go there, day or night, inside is always packed full. This is the place to be seen. Every IT girl, celebrity, person of note in the world is inside.
- I’m speaking of cause about club VALID. Now I know all of you have been turned away at some point or another, so today I’m going to share with you the unwritten rules about what to do….
- When waiting in line...
- When facing the bouncer
- When your ticket is checked
- And once you're in
- how to get to the VIP area
- To make valuable connections
- No-one likes waiting in line and so emerged a concept of crawl budget optimisation.
- The theory is that search engines don’t have unlimited resources to crawl every page every day. So they allocate a crawl budget to each site. This is the number of URLs search engines can and want to crawl. For years the SEO community have feared that if you don’t take steps to optimise your crawl budget you wouldn’t have enough.
- It got to a point where we were trying to come up with a precise way to calculate crawl budget with data from the crawl stats report, which is a woefully incorrect picture as this is an aggregate of 12 different google crawlers - not only googlebot. But more importantly, it doesn’t say anything about which pages are being crawled when.
- To understand if your site crawl budget issues check in the Google Search Console Coverage report for the exclusion “Discovered - Currently not indexed”. For any URLs with that classification you can easily jump to the front of the crawl queue - just get your name on the guest list.
- The best way to do this is to link up with friends who are respected by the club who can vouch for you. This can be part of your internal link infrastructure or as an external link. Bear in mind the goal isn’t building link equity, it’s for rapid discovery by googlebot, so an external link can be as simple as including the URL in a Tweet.
- The second best way is to call ahead and book a place by adding the URL in an XML sitemap.
- Think of your XML sitemaps as a list of SEO relevant URLs you recommend search engines crawl as soon as page is added or updated, which isn’t necessarily every single page of your website. Used wisely it can help search engines crawl your site more intelligently. Used incorrectly, it will draw attention to your website’s flaws and you will see errors like these begin to appear in your coverage report.
- We all know queues are rarely first come first serve. Those who are known to bring value to the club, like regular patrons or celebrities, can jump the line. If you aren’t one of those pages, you don’t know how long you will have to wait. It could be an hour, a week, a few months or even longer. When your urls will be crawled is impacted by site architecture patterns, the URL history, the domain reputation and of cause, whether they have links and if they are included in your XML sitemaps.
- You can influence which URLs should be next in line by include the last modification date and time. This informs search engines when the page last significantly changed and if it should be recrawled. And it’s because of this tag that it’s important XML sitemap be dynamic and instantly update along with your website. I’ve seen many XML sitemaps coded to update only once per day. Essentially, you are choosing to wait in line for 24 hours.
- What is clear is that with a small amount of effort it is ridiculous easy to get on the guest list & induce googlebot to crawl a page, even within a short timeframe. The concept of a need for crawl budget optimisation is nothing but hot air - there is no substance behind it.
- The true problem facing many sites is the exact opposite of crawl budget issues. Often Googlebot finds too many pages. Undesirable URLs that are not contained in your sitemap. Orphaned URLs not linked on your website. Bizarre URLs you may not even have known exist. Focus on which URLs googlebot crawls when and what impact this has on how search engines understand site architecture, on the speed of indexing and the quality of what is seen by searchers.
- Because if your coverage report look something like this, your problem is not crawl budget. It’s site architecture. These URLs have been denied entry to club VALID.
- But unlike at most clubs, the bouncer has told you why and I’m going to tell you what to do about it.
- Let’s start with the basics. To some extent you can rely on your reputation if you are a big name. But if you walk up to the bouncer without pants on, chances are they wont let you in. You need to meet the dress code.
- If you are wearing 5xx errors go home and improve your server infrastructure. If you are wearing a 404 page not found, the prevailing advice is to fix the perceived error, else you will waste cawl budget on such pages. Lets get 2 thing straight, one, Fuck crawl budget...and two, the 404 is not an error, its an exclusion. If the page truly doesn’t exist because it was intentionally removed, there is nothing wrong.
- There is no Google penalty for amassing 404 codes - that is a myth. But that’s not to say they are best practice. If the URL had any ranking signals, these are lost to the 404 void.
- So it’s common in SEO to implement 301 redirects when a page is removed.
- In which case googlebot would crawl the original URL, see the 301 status code, and then add the destination URL to the crawl queue. The ranking signals with no dilution will be passed once google crawls and confirms that the destination URL has similar content.
- But if you redirect to an irrelevant page, such as the homepage, Google will treat this as a soft 404 and won’t pass on the ranking signals. I can’t definitively tell you why this is, but I suspect it’s two fold. Firstly, for user experience, if i click on a search result expecting to land on a specific piece of content and all of a sudden I am on the homepage with no explanation that is not a better user experience than a custom 404 page where at least the user understands what happened. And secondly, rather than trading a 301 exclusion for a soft 404, encourage SEOs to signal to Google to rapidly de-index such content using a 410 code.
- The main takeaway here is that there is no inherently good or bad codes but there are right and wrong codes for specific circumstances and right ways and wrong ways to implement those codes.
- Like with redirects. The rules are clear. They won’t be allowed in to the club if they come in a big group.
- If you have redirected page 1, to page 2 and then later 2 to page 3 you are creating a redirect chain - which is bad enough as it takes additional time to follow and forward the ranking signals, causes unnecessary load on your servers and latency for users. But if this continues past 5 pages, Googlebot gives up. You’re told “get to the back of the line”. So whatever the destination page was going to be, it won’t benefit from the transfer of ranking signals, and it may not even be crawled to get into the index.
- Just because it is named a permanent redirect, doesn’t mean it should live forever. Break the chains and redirect each page to the final destination directly.
- While we are n the topic of SEO misconceptions, let’s also tackle duplicate content. To many people believer that all duplicate content is bad. This is absolutely absurd. There are perfectly legitimate reasons to have duplicate content on site - for example if you are utilizing AMP, or you track using UTM parameters or you have a sort function to change the order of products. Duplicates don’t necessarily need to be redirected or removed but rather handled gracefully - and there are many options by which to achieve this.
- One way is with a rel=canonical tag. An entry stamp that signals to search engines which of the duplicate URLs you wish to be indexed. If the tag is accepted, the alternate pages will be crawled, but much less frequently. They will excluded from the index, passing their ranking signals on to the canonical. So when you see the “alternate page with proper canonical tag” exclusion, it means your canonical has been accepted. This is a good thing.
- If you see duplicate without user-selected canonical. Either you haven’t added the tag or it’s implemented incorrectly.
- The worst problems arise when you see exclusions such as these, which show that Google thinks your trying to game the system and transfer signals to pages that don’t deserve them. In which case Google will happily ignore your stamp and make its own decision. Mis-signaling can be due to sitemaps or internal link infrastructure not prioritising the canonical or from using canonicals incorrectly, like on pages that aren’t actually duplicates.
- The second option for handling duplicate content is by telling Googlebot not to crawl within Google Search Console URL parameter handling. When you specify to crawl no URL, those pages don’t exist as far as Google is concerned. Googlebot won't crawl the URLs, saving load on your server. But that comes at a price, if Googlebot can’t crawl, Caffeine can’t process signals - which may impact ranking - or extract internal links to add to the crawl queue - which may slow down site indexing.
- Another way to stop a crawler is by disallowing the URL in the robots.txt file.
- It’s the digital equivalent of a “no entry” sign on an unlocked door. And while googlebot obeys these instructions, it does it to the letter of the law, not the spirit.
- So you may have pages that are specifically disallowed in robots.txt showing up in the search results.
- Because if a blocked page has other strong ranking signals, Google may deem it relevant to index. Despite not having crawled the page. But because the content of that URL is unknown to Google, the search result looks something like this.
- To definitively block a page from appearing in SERPs, you need to use a “noindex” robots meta tag or X-Robots-Tag in the HTTP header.
- One problem is, if the tag it’s present for a long time, it will eventually lead Google to nofollow the page’s links as well - which means they won’t add those links to the crawl queue and ranking signals won’t be passed to linked pages.
- And you start to see that all of this is rather complicated. And if one signal contradicts another, it’s not always clear how search engines will respond. So rather than using all these directives and hints to bandaid together your website, take a step back, breathe, and then work on the architecture of your website so that you can minimise robots directives as much as possible.
- And move onto the next problem “Crawled - currently not indexed”. Essentially, the bouncer has looked you up and down and said ‘not tonight’. This is most commonly due to quality issues - thin content, doorway pages, user-generated spam, anything that is clearly not worthy of indexing will be rejected. But if content is worthy but not being indexed you’re likely being tripped up by rendering.
- Let’s talk about JavaScript - your ticket to a better user experience and more challenging SEO.
- Of cause both Google and Bing are capable of indexing JS-generated content - because both use evergreen headless Chromium. Google shows the necessary rendering as a rather simple process - HTML goes into a render queue, it is then rendered and sends the DOM to be indexed. But Javascript injects a deeper level of complexity into the indexing equation.
- Because there are two waves of indexing whenever javascript is involved - a bouncer who checks your ticket at the door followed by another who checks you bag just inside. The first wave indexes a page based on the initial HTML from the server. This is what you see when you right click & view page source.
- The second indexes based on the DOM, which includes both the HTML and the rendered JS from the client side. This is what you see when you right click & inspect.
- The challenge is that the 2nd wave of indexing is deferred until Caffeine has the rendering resources available. This means it takes longer to index Javascript-reliant content than HTML only content. Anywhere from days up to a few weeks from the time it was crawled.
- But unlike most things in technical SEO, there is a clear solution. Use server side rendering so that all essential content is present in the initial HTML, allowing search engine to index it immediately. This should include your hero SEO elements like page titles, headings, canonicals, structured data and of course your main content and links.
- To understand if you content is rendered on the server side you can’t rely on the mobile friendly test tool - it doesn't use the same pipeline that a true rendering engine would.
- The live test functionality in the URL inspection tool is a bit better as it can tell you if Google is technically able to render the page.
- But it is also significantly more patient. The real Caffeine may not be able to index the full content because of timeouts. So be sure you adhere to the 5 second render rule - as Caffeine will tend not to wait more than 5 seconds for a script when indexing.
- To truly understand the outcome of rendering, you can’t rely on these tools. You need to take a close look at your ticket and make sure you understand what all the text and codes mean. Otherwise you have no idea whether your ticket is valid. You have to understand your rendering stack as it directly impacts the SEO performance of your site.
- No critical content should be reliant on the render as the time to index could be weeks slower. That’s weeks worth of having to justify investment in a strategy that seemingly isn’t performing to clients. Weeks worth of lost sales to competitors. Weeks worth of waiting on rankings for time-limited URLs that are likely to be outdated before they’re indexed.
- But then you're inside. And you will see that it is always packed full. But some of the people in the crowd aren't what you expected.
- Like these guys. They got inside. They are having a good time. But do they have the slightest chance of converting anyone that night? No way. Problem is, they used your name to get in. Their behaviour reflects on your reputation.
- Because URLs are not ranked solely on their own merits, but also on the company the keep and the family they belong to. Every page indexed by search engines impacts how the quality algorithms evaluate your domain reputation.
- If you have a lot of URLs that aren’t in your sitemap, but are in the index, you have a problem. Either your sitemap doesn’t include all your SEO relevant URLs. Which can be easily fixed. Or you are suffering from index bloat where an excessive number of low-value pages have made it into the index. This it’s commonly caused by auto-generated pages - like filter combinations, archive pages, tag pages, user profiles, pagination, rogue parameters, you get the gist.
- Your goal shouldn’t be to get as many pages into the index as possible. It’s already crowded. You will not get seats for a big group - you would be lucky to get two seats or even one. And with so many pages competing for the same search intent, it becomes confusing to search engines which pages deserve to be ranked. You will have more success if you combine their signals and have one attractive page which can stand out in the crowd.
- But your goal shouldn’t be to rank as many pages as possible. If your ranking lead to a bad user experience because they land on a low quality page and bounces, that is hurting your brand.
- Your goal should be to make a valuable match between the user and the page. So make sure you know all the types of pages you have in the index and only put the best face of your brand forward. No bloat. No noise. Only pages you want your users to land on.
- There are many mechanisms by which you can eject unwanted pages out of the index. If the page has zero value, I recommend you send a 410 code as search engines will know you intended to remove the content and swiftly de-index it. If it has value and merging with a desirable page is an option, then 301. If there is a legitimate reason for duplicate content, set the canonical. If there's not a good reason, an it’s run through a parameter, use URL parameter handling. And if none of the above is an option, resort to a noindex tag. NOT robots.txt disallows as this is not guaranteed to deindex the page.
- And after you finish the cleanup, have a drink and celebrate because your pages will be running with the IT girls.
- But there is a difference between an ‘it’ girl and the indexable elite.
- Everyone who is deserving will eventually get into club VALID. But nobody wants to spend half their night waiting in line. Getting in fast is a competitive advantage in SEO. The sooner your high-quality pages are indexed, the sooner they can start establishing top spot ranking in search results. This is even more critical when its comes to time-sensitive content, like breaking news, or for pages with a short lifespan, like product listings. So it’s worthwhile to invest the time to become a VIP.
- Because they have their own entryway, prioritised over all other URLs. Use can use this entry if you manually submit a URL in Google Search Console or Bing Webmaster Tools. But this method has its limitations with Google, who only allows 50 such request per day.
- If you need scale, the better option is to integrate with their new APIs which allow you to directly notify the search engines as soon as a URL is added, updated or removed.
- You can expect a crawl within 1 minute of submitting any URL via the API. And yes, I do mean any submitted URL. No matter the content, even though officially Google says the API is only there to support pages with job postings and livestream structured data. Of course I’d never advise you to submit anything against the Google guidelines, but if developer happen to accidentally submit a different content type, I’m simply pointing out that it would be crawled by googlebot with priority.
- Bing is doubling down on attracting VIP content. They have said in a few years they hope not to rely on traditional systematic web crawling but rather on the submission of URLs either manually in Bing Webmaster Tools or automated with the API.
- In a shocking divergence from their usual open nature, Google hasn’t stated their plans. But allusive comments made by Gary, the non-content-specific naming of the API and the fact that it will crawl any submitted URL, no matter the content, invites speculation as to the company’s plans for its future.
- But right now, if you use the search engine APIs for priority entry and server side rendering for speedy indexing you can get up and ranking in no time.
- But it get even better. Submitting with the Google API not only gets you VIP entry, but also admission to the VIP areas - the rich results. They can offer this because complete and correct structured data is compulsory for all VIPs. And so they have no problem refactoring the content for presentation on their platforms.
- And entry to the VIP areas is critical to success. Because for competitive terms on mobile, ranking in organic position 1 really doesn’t mean much anymore. If you want to rank for boris johnson, forget about 10 blue links.
- Because only 8 make it onto the first page. And frankly, if you’re not in position 1 or 2 you’re not getting much visibility. And that is just in google search.
- Indexing speed is a critical factor whether you will be featured within Google Discover as much of the content is time-limited.
- Indexing speed is critical to be featured in Google News for the same reason, Google wants the content to be fresh.
- And now we have Google for Jobs - where schema markup and the API combine together to index, format and aggregate listings for jobseekers. Essentially this replaces the functionality of category pages on job sites, passing the user over only for conversion.
- And so we start to see that search engines are shifting from keyword to topics, from answers to journey, from queries to feeds and most importantly from being a traffic conduit to a content hub. With these types of interactive, rich experiences Google is going deeper into the user journey. Some SEOs see it as an opportunity others as a threat. But no matter what side you are on, the reality this is just the beginning.
- If Google officially opens up the Indexing API for more content types, more VIP experiences like Google for Jobs will be created. And our world will change. Because Google will become the distribution platform for your whole content, not for only a snippet of it, and if you refuse to integrate, that will impact your ability to reach and engage your target audience.
- Maybe we need to rethink our KPIs and strategies. It’s not about crawl budget, or indexing rate, or rank tracking or even organic users. I believe that as SEOs we need to focus on what drives value for brands - whether this is revenues or conversions or some other KPI. If by giving up some control of your content to search engines, you end up driving more value for your brand, well... you win.
- So - here are the top 5 takeaways from this talk. 1. Crawl budget optimisation is bullshit - it’s easy to get a page crawled, simply use the most relevant one of these 4 methods.
- 2. SEO hacks tend to be exactly that - hacky. You want to be presentable when you reach the Googlebot bouncer - Know how the different directives are processed by search engines. Don’t send conflicting signals. And if you get turned away stop and listen because the bouncer tells you why!
- 3 SEO can fail despite onpage and offpage being on point because of the processes behind the rank, especially rendering strategies. You need to understand your rendering stack.
- 4. Just because a URL is valid, doesn’t mean it deserves to be. Know what pages you have in the index and how this impacts your brand reputation.
- And finally, aim to achieve VIP status because search engines are evolving into platform ecosystems and if you don't contribute content, you will be left out in the cold.
- And that my friends, is everything you need to know to enjoy yourself at club VALID.