GPTBot
OpenAI's crawler used to collect publicly available web content that may be used to improve future models; controllable via robots.txt.
GEO AI
OAI-SearchBot
OpenAI's crawler that builds the search index used to surface and link websites in ChatGPT search results; distinct from the training crawler GPTBot.
In full
OAI-SearchBot exists so publishers can participate in ChatGPT's search and citation experience without allowing training use of their content. Disallowing it removes a site from those results in the same way disallowing Googlebot removes it from Google. Confusing the two agents is the most common self-inflicted AI visibility mistake: a blanket block of every OpenAI user agent forfeits citations as well as training.
Example
A news publisher disallows GPTBot but explicitly allows OAI-SearchBot so its articles can still be linked in ChatGPT answers.
Related terms
ChatGPT-User
OpenAI's agent that fetches a specific page in real time because a user or an agentic task requested it, rather than for indexing or…
AI crawler
An automated agent operated by an AI company to fetch web content for model training, search indexing, or live retrieval when answering a…
Citation share
The proportion of AI answer citations across a tracked prompt set that point to a given domain, used as the AI-era analogue of ranking…
robots.txt
A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.
Where to read more
- Overview of OpenAI Crawlers · Documentation · developers.openai.com