Robots meta tag
A `<meta name="robots">` element in the HTML head carrying per-page indexing and serving directives such as noindex, nofollow, nosnippet…
Crawling Indexing
X-Robots-Tag
An HTTP response header that carries the same robots directives as the meta robots tag, usable for any file type including PDFs and images.
In full
Because it lives in the response header rather than the document body, X-Robots-Tag is the only way to apply noindex, nosnippet or noimageindex to non-HTML resources. It can be set globally in server config or per-response by the application, and can be scoped to a named crawler with a prefix, e.g. `X-Robots-Tag: googlebot: noindex`. It is also a convenient way to noindex entire directories without editing templates.
Example
An Nginx location block adds `add_header X-Robots-Tag "noindex";` to every `/invoices/*.pdf` response.
Related terms
noindex
A robots directive, delivered by meta tag or X-Robots-Tag header, that instructs search engines to drop a page from their index.
robots.txt
A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.
HTTP status codes
Numeric codes in an HTTP response that tell crawlers how to treat a URL: 200 index it, 3xx follow the redirect, 404/410 drop it, 5xx come…
nosnippet / max-snippet
Robots directives controlling whether and how much of a page's text a search engine may show as a snippet or use in generated summaries.
Where to read more
- Block Search Indexing with noindex · Documentation · developers.google.com
- Robots Meta Tags Specifications · Documentation · developers.google.com