AI Overviews
Google's AI-generated summaries shown at the top of many search results, synthesising information from multiple web pages with links to…
GEO AI
Google-Extended
A robots.txt control token that governs whether a site's content can be used to improve Gemini models and grounded Gemini API responses, without affecting Google Search.
In full
Google-Extended is not a separate crawler — it is a token applied to content already fetched by Googlebot, letting publishers opt out of certain generative AI training and grounding uses while remaining fully in Search. Crucially, disallowing it does not remove a site from AI Overviews or AI Mode, which are Search features governed by ordinary snippet controls and the Search generative AI control. Confusing these two mechanisms is a widespread error.
Example
`User-agent: Google-Extended` / `Disallow: /` opts a publisher out of Gemini training while its pages continue appearing in AI Overviews.
Related terms
Search generative AI control
A Search Console setting letting site owners include or exclude their site from Google's generative AI features — AI Overviews, AI Mode…
nosnippet / max-snippet
Robots directives controlling whether and how much of a page's text a search engine may show as a snippet or use in generated summaries.
robots.txt
A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.
Model training opt-out
Mechanisms by which a publisher signals that its content should not be used to train AI models, principally robots.txt tokens and…
Where to read more
- Google Crawler (User Agent) Overview · Documentation · developers.google.com