Google-Extended crawler control introduced
Google introduced a robots.txt token letting publishers opt out of having crawled content used to train Gemini and Vertex AI generative models.
What changed
Google-Extended is a robots.txt user-agent token rather than a distinct crawler: blocking it stops content Googlebot has already fetched from being used to train and ground Google's generative AI models. Google was explicit that it has no effect on inclusion or ranking in Google Search. Publishers noted the asymmetry, since opting out of AI training did not at that point allow opting out of AI Overviews, which are generated from search-indexed content.
Timing
Rollout began 28 September 2023. No completion date was announced. Rankings typically remain unstable for the whole rollout window, so measurements taken inside it are not a reliable read on the outcome.