noindex
A robots directive, delivered by meta tag or X-Robots-Tag header, that instructs search engines to drop a page from their index.
Crawling IndexingRobots Exclusion ProtocolREP
robots.txt
A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.
In full
robots.txt controls crawling, never indexing: a disallowed URL can still appear in results as a URL-only listing if other pages link to it. Rules are grouped by user-agent token, and the most specific matching group wins; Google supports wildcards (*) and end-of-string anchors ($). It applies per protocol, host and port, and Google caches it for roughly 24 hours. A 5xx response on robots.txt causes Google to pause crawling the host.
Example
`User-agent: *` / `Disallow: /cart/` blocks a checkout path, while a separate `User-agent: GPTBot` / `Disallow: /` group blocks OpenAI's training crawler.
Related terms
Crawl budget
The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.
GPTBot
OpenAI's crawler used to collect publicly available web content that may be used to improve future models; controllable via robots.txt.
Google-Extended
A robots.txt control token that governs whether a site's content can be used to improve Gemini models and grounded Gemini API responses…
URL parameters
Key-value pairs after a question mark in a URL; they frequently create duplicate content and crawl waste when they do not change page…
Where to read more
- About Applebot · Documentation · support.apple.com
- ai.robots.txt Community Blocklist · Dataset · github.com
- ai.txt by Spawning · Standard · site.spawning.ai
- Block Search Indexing with noindex · Documentation · developers.google.com
- Brave Search Crawler Documentation · Documentation · search.brave.com
- CCBot Crawler Documentation · Documentation · commoncrawl.org