GPTBot
OpenAI's crawler used to collect publicly available web content that may be used to improve future models; controllable via robots.txt.
GEO AIAI training controls
Model training opt-out
Mechanisms by which a publisher signals that its content should not be used to train AI models, principally robots.txt tokens and contractual or technical blocking.
In full
Opt-out is voluntary and vendor-specific: robots.txt entries for GPTBot, Google-Extended, ClaudeBot and others are honoured by the vendors that publish them, but the file has no legal force and non-compliant scrapers ignore it. Stronger enforcement requires CDN-level blocking, authentication, or licensing agreements. Publishers should distinguish training use from retrieval use before blocking, because the two have opposite commercial consequences.
Example
A media group blocks training agents in robots.txt, enforces with edge rules, and separately licenses its archive to two model providers.
Related terms
Google-Extended
A robots.txt control token that governs whether a site's content can be used to improve Gemini models and grounded Gemini API responses…
ClaudeBot
Anthropic's family of web crawlers and fetchers, including a general crawler and agents that retrieve pages for search results and for…
AI crawler
An automated agent operated by an AI company to fetch web content for model training, search indexing, or live retrieval when answering a…
AI content licensing
Commercial agreements in which publishers license their content to AI companies for training, retrieval or attribution, as an alternative…