Crawl budget
The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.
Crawling IndexingServer log analysis
Log file analysis
Examining raw server access logs to see exactly which URLs crawlers fetched, how often, with what status codes and response times.
In full
Logs are the only ground truth about crawler behaviour — everything else is inference. They reveal crawl budget waste on parameter URLs, sections crawlers never reach, spikes of 5xx errors, redirect chains, and which bots are consuming bandwidth. Verification by reverse DNS matters because user-agent strings are trivially spoofed. In 2026 logs are also the primary way to measure AI crawler traffic from agents such as GPTBot, ClaudeBot and PerplexityBot.
Example
Logs show Googlebot spent 62% of its monthly fetches on `?sort=` URLs, which are then disallowed in robots.txt.
Related terms
Googlebot
Google's main web crawler, which fetches pages for Google Search using two user agents: Googlebot Smartphone and Googlebot Desktop.
GPTBot
OpenAI's crawler used to collect publicly available web content that may be used to improve future models; controllable via robots.txt.
HTTP status codes
Numeric codes in an HTTP response that tell crawlers how to treat a URL: 200 index it, 3xx follow the redirect, 404/410 drop it, 5xx come…
Orphan page
A page with no internal links pointing to it, reachable only via sitemap, external link or direct URL.