OAI-SearchBot
OpenAI's crawler that builds the search index used to surface and link websites in ChatGPT search results; distinct from the training…
GEO AI
GPTBot
OpenAI's crawler used to collect publicly available web content that may be used to improve future models; controllable via robots.txt.
In full
OpenAI documents GPTBot alongside OAI-SearchBot, which builds the search index used to surface and link sites, and ChatGPT-User, which fetches a page because a user or agent asked for it. Blocking GPTBot opts content out of training-oriented crawling without necessarily removing a site from ChatGPT's search results, since those use different agents. OpenAI publishes IP ranges so operators can verify requests rather than trusting the user-agent string.
Example
`User-agent: GPTBot` / `Disallow: /` blocks training collection while `OAI-SearchBot` remains allowed for search visibility.
Related terms
ChatGPT-User
OpenAI's agent that fetches a specific page in real time because a user or an agentic task requested it, rather than for indexing or…
AI crawler
An automated agent operated by an AI company to fetch web content for model training, search indexing, or live retrieval when answering a…
Model training opt-out
Mechanisms by which a publisher signals that its content should not be used to train AI models, principally robots.txt tokens and…
robots.txt
A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.
Where to read more
- Overview of OpenAI Crawlers · Documentation · developers.openai.com