AI crawlers, decoded

The four things a bot can be

Search-index crawlers build the corpus a product retrieves from when it answers. Blocking one of these removes you from that product's answers and citations — 19 of the agents below fall into this category or the next one, and blocking them has a direct visibility cost.

User-triggered fetchers retrieve a single page because a person asked. Blocking them means a user who explicitly pastes your URL into an assistant gets nothing back. That is rarely what anyone wants.

Training crawlers collect corpora for model training. Blocking them is a licensing and policy decision, not an SEO one, and it does not remove you from that vendor's search product.

Control tokens such as Google-Extended and Applebot-Extended are not crawlers at all. They are robots.txt names that govern how already-crawled content may be used downstream. Disallowing them changes usage rights, not crawling, and does not affect search inclusion.

The rule most sites actually want

Allow retrieval and user-triggered agents, decide on training deliberately. Something like this expresses "index and cite me, don't train on me" — adjust to your own policy rather than pasting it blind:

# Allow retrieval so we stay citable
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Opt out of training corpora
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# Usage control tokens (not crawlers)
User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Note what is deliberately absent: Googlebot and bingbot are untouched. Blocking those to "stop AI" would remove you from Google and Bing themselves, including AI Overviews, which are grounded in the ordinary search index rather than crawled by a separate agent.