Crawl budget
The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.
Crawling Indexing
Crawl demand
How much a search engine wants to crawl a set of URLs, driven by their popularity, how often they change, and the site's perceived overall quality.
In full
Even when a server could serve far more requests, Google will not crawl URLs it does not value. Demand rises for URLs that are linked to, that appear in sitemaps with plausible lastmod dates, that historically change, and that sit on sites Google rates highly. Demand collapses for stale, duplicative or low-quality URL spaces — which is why crawl problems on large sites are usually quality problems, not server problems.
Example
A blog that publishes daily sees its homepage crawled hourly while a five-year-old tag archive is crawled twice a year.
Related terms
Crawl rate limit
The maximum simultaneous connections and fetch frequency a crawler will use for a host, adjusted automatically based on server response…
Freshness systems (QDF)
Google's systems that boost recent content for queries where recency matters, such as breaking news, sports scores or evolving topics.
Index bloat
A condition where a site has far more URLs indexed or crawlable than it has genuinely useful pages, diluting crawl attention and…
XML sitemap
An XML file listing URLs a site wants crawled, optionally with lastmod dates, used as a discovery aid by Google, Bing and other engines.
Where to read more
- Crawl Budget Management for Large Sites · Documentation · developers.google.com