Crawl budget
The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.
Crawling IndexingCrawl capacity limit
Crawl rate limit
The maximum simultaneous connections and fetch frequency a crawler will use for a host, adjusted automatically based on server response times and errors.
In full
Google raises crawl capacity when a host responds quickly and lowers it when responses slow down or return 5xx and 429 errors. The old manual crawl-rate setting in Search Console was retired; capacity is now managed automatically, though site owners can signal overload with 503/429 responses. Because capacity is per host, sharing infrastructure across many hostnames can concentrate or dilute crawling.
Example
After a database issue pushes median TTFB from 200 ms to 3 s, Googlebot's daily fetches drop by half until performance recovers.
Related terms
Crawl demand
How much a search engine wants to crawl a set of URLs, driven by their popularity, how often they change, and the site's perceived overall…
Time to First Byte
The interval between a navigation request starting and the first byte of the response arriving; a foundational input to LCP and a signal…
HTTP status codes
Numeric codes in an HTTP response that tell crawlers how to treat a URL: 200 index it, 3xx follow the redirect, 404/410 drop it, 5xx come…
Log file analysis
Examining raw server access logs to see exactly which URLs crawlers fetched, how often, with what status codes and response times.