Mastering Crawl Budget: Get Your Important Pages Crawled

Crawl budget is the number of URLs a search engine can and wants to crawl on your site. Most small and medium sites never need to worry about it. It matters when you have a very large site, content that changes quickly, or many pages sitting in “Discovered – currently not indexed”.

Here’s the short version. For the full treatment, read our definitive guide to crawl budget.

The two halves of crawl budget

  • Crawl capacity: how much crawling your server can take. Fast, error-free responses raise it, and errors lower it.
  • Crawl demand: how much a search engine wants your URLs, based on their popularity, how fresh they are, and how many it thinks you have.

What helps

  • Keep your sitemap to canonical, indexable URLs with accurate lastmod dates.
  • Block low-value URL patterns, such as faceted navigation and internal search, in robots.txt. Don’t rely on noindex for this, because Google still has to request the page to see the tag.
  • Remove or redirect dead pages, shorten redirect chains and consolidate duplicates.
  • Improve server response times.

Handling URL parameters

Google retired Search Console’s URL Parameters tool in 2022, so parameters now need handling on your site, with canonical tags, robots.txt rules and consistent internal links.

How to tell if it’s working

The Crawl Stats report in Search Console shows requests, response codes and server response times. The Page indexing report shows what became of those requests. Your server logs give the most precise view, and we cover how to use them in our guide to log file analysis.


Posted

in

by