AI crawler
An automated agent operated by an AI company to fetch web content for model training, search indexing, or live retrieval when answering a…
GEO AIClaude-WebClaude-UserClaude-SearchBot
ClaudeBot
Anthropic's family of web crawlers and fetchers, including a general crawler and agents that retrieve pages for search results and for individual user requests.
In full
Anthropic documents its user agents and publishes IP ranges for verification, and states its crawlers respect robots.txt directives. As with OpenAI, the agents serve different purposes — general crawling, search indexing to support citations, and user-initiated retrieval — so a site owner should decide per agent rather than blocking the family. Blocking search-oriented agents removes the site from citation in Claude's answers.
Example
A documentation site allows all Anthropic agents and sees ClaudeBot fetches spike after a release announcement.
Related terms
GPTBot
OpenAI's crawler used to collect publicly available web content that may be used to improve future models; controllable via robots.txt.
PerplexityBot
Perplexity's crawler for building its search index, alongside a separate user-triggered agent that fetches pages in response to specific…
Model training opt-out
Mechanisms by which a publisher signals that its content should not be used to train AI models, principally robots.txt tokens and…
robots.txt
A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.
Where to read more
- Does Anthropic Crawl Data from the Web, and How Can Site Owners Block the Crawler? · Documentation · support.claude.com