Crawl budget
The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.
Crawling IndexingInfinite spaceSpider trap
Crawl trap
A URL pattern that generates effectively unlimited unique URLs, causing crawlers to loop indefinitely and exhaust crawl budget.
In full
Classic examples are calendars with next-month links extending forever, relative-path bugs producing `/a/b/a/b/a/`, session ids in URLs, and unbounded filter combinations. Traps are visible in logs as crawl volume rising without indexation rising. Fixes include robots.txt patterns, nofollow on generative controls, bounding the generated range, and returning 404 outside sensible limits.
Example
An events calendar lets Googlebot walk to the year 2147 one month at a time, generating hundreds of thousands of empty pages.
Related terms
Faceted navigation
Filter and sort controls on listing pages that can generate a combinatorial explosion of crawlable URLs from a small product set.
Index bloat
A condition where a site has far more URLs indexed or crawlable than it has genuinely useful pages, diluting crawl attention and…
robots.txt
A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.
Log file analysis
Examining raw server access logs to see exactly which URLs crawlers fetched, how often, with what status codes and response times.