Crawl rate limit
The maximum simultaneous connections and fetch frequency a crawler will use for a host, adjusted automatically based on server response…
Crawling Indexing
Crawl budget
The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.
In full
Crawl budget is the intersection of what a host can serve without degrading (crawl capacity limit) and what the search engine wants (crawl demand, driven by popularity, staleness and perceived quality). Google states that most sites under a few thousand URLs never need to think about it; it becomes a real constraint for large e-commerce, classifieds, and sites generating many low-value URLs. The practical levers are removing duplicate and faceted URL spaces, fixing slow responses, and returning accurate status codes.
Example
A marketplace with 4 million filter-generated URLs sees Googlebot spend 70% of its fetches on parameter URLs, starving new product pages of crawls.
Related terms
Crawl demand
How much a search engine wants to crawl a set of URLs, driven by their popularity, how often they change, and the site's perceived overall…
Faceted navigation
Filter and sort controls on listing pages that can generate a combinatorial explosion of crawlable URLs from a small product set.
Log file analysis
Examining raw server access logs to see exactly which URLs crawlers fetched, how often, with what status codes and response times.
Index bloat
A condition where a site has far more URLs indexed or crawlable than it has genuinely useful pages, diluting crawl attention and…
Where to read more
- Crawl Budget Management for Large Sites · Documentation · developers.google.com