SEO Atlas
GEO AIAI botLLM crawler

AI crawler

An automated agent operated by an AI company to fetch web content for model training, search indexing, or live retrieval when answering a user's question.

In full

These fall into three functional classes with different consequences: training crawlers, search-index crawlers that build a retrievable index, and user-triggered fetchers that retrieve a page in response to a specific request. Blocking them is a business decision, not a technical one — blocking a training crawler protects content but blocking a retrieval agent removes you from answers. Most declare a user-agent token and respect robots.txt, and several publish IP ranges for verification; bot-management rules at the CDN can block them regardless of robots.txt.

Example

A publisher allows OAI-SearchBot and ClaudeBot's search fetches while disallowing training-oriented agents in robots.txt.

Related terms

GPTBot

OpenAI's crawler used to collect publicly available web content that may be used to improve future models; controllable via robots.txt.

ClaudeBot

Anthropic's family of web crawlers and fetchers, including a general crawler and agents that retrieve pages for search results and for…

PerplexityBot

Perplexity's crawler for building its search index, alongside a separate user-triggered agent that fetches pages in response to specific…

Google-Extended

A robots.txt control token that governs whether a site's content can be used to improve Gemini models and grounded Gemini API responses…

Model training opt-out

Mechanisms by which a publisher signals that its content should not be used to train AI models, principally robots.txt tokens and…

Where to read more

Tools that touch this

Prerender.io