GPTBot crawler introduced
OpenAI published a named crawler and robots.txt token so publishers could opt out of having their content used for model training.
What changed
OpenAI documented GPTBot as the user agent used to gather web content for training generative models, and published IP ranges plus robots.txt guidance for blocking it. Disallowing GPTBot signals that a site's content should not be used in foundation model training. It was the first widely adopted opt-out mechanism for AI training crawls and prompted large numbers of publishers to add AI-specific robots.txt rules. OpenAI later added separate agents for search indexing and user-initiated fetches.
Timing
Rollout began 7 August 2023. No completion date was announced. Rankings typically remain unstable for the whole rollout window, so measurements taken inside it are not a reliable read on the outcome.