noindex
A robots directive, delivered by meta tag or X-Robots-Tag header, that instructs search engines to drop a page from their index.
Keeping cart, checkout, account and order confirmation URLs out of the index, since they have no search value and can leak session data.
These URLs are typically parameterised, personalised and infinite in variety, so they consume crawl budget and generate index bloat. The correct treatment is noindex on the pages plus, where the space is unbounded, a robots.txt disallow — remembering that a disallowed URL cannot be seen to be noindexed. Order confirmation pages occasionally get indexed with customer data, which is a privacy incident as well as an SEO problem.
A site finds 1,200 order confirmation URLs indexed with order references, and applies noindex plus authentication.
A robots directive, delivered by meta tag or X-Robots-Tag header, that instructs search engines to drop a page from their index.
A condition where a site has far more URLs indexed or crawlable than it has genuinely useful pages, diluting crawl attention and…
A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.
Key-value pairs after a question mark in a URL; they frequently create duplicate content and crawl waste when they do not change page…
The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.