SEO Atlas
Crawling Indexing

noindex

A robots directive, delivered by meta tag or X-Robots-Tag header, that instructs search engines to drop a page from their index.

In full

noindex is a directive, not a hint: compliant crawlers will remove the URL once they re-crawl and see it. Critically, the page must remain crawlable for the directive to be seen — blocking a URL in robots.txt prevents the crawler from ever reading the noindex, which is the single most common reason unwanted pages stay indexed. Google no longer supports a noindex directive inside robots.txt.

Example

An internal search results template ships `<meta name="robots" content="noindex, follow">` so the pages are dropped but their outgoing links are still crawled.

Related terms

Robots meta tag

A `<meta name="robots">` element in the HTML head carrying per-page indexing and serving directives such as noindex, nofollow, nosnippet…

X-Robots-Tag

An HTTP response header that carries the same robots directives as the meta robots tag, usable for any file type including PDFs and images.

robots.txt

A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.

Index bloat

A condition where a site has far more URLs indexed or crawlable than it has genuinely useful pages, diluting crawl attention and…

Indexing

Storing and organising a crawled page in a search engine's index so it becomes eligible to be retrieved and ranked for queries.

Where to read more