Canonicalization
The process by which a search engine groups duplicate or near-duplicate URLs into a cluster and picks one representative URL to index and…
Crawling IndexingQuery strings
URL parameters
Key-value pairs after a question mark in a URL; they frequently create duplicate content and crawl waste when they do not change page content.
In full
Parameters fall into two classes: content-changing (a filter value, a page number) and non-content-changing (session ids, tracking, sort order, view mode). Google's URL Parameters tool in Search Console was retired in 2022, so control now rests entirely with the site: consistent parameter ordering, canonical tags, robots.txt patterns, and avoiding parameters in internal links. Tracking parameters should be stripped from internal navigation and sitemaps.
Example
`?utm_source=newsletter` on internal links causes Google to crawl and cluster thousands of tracked duplicates of the homepage.
Related terms
Faceted navigation
Filter and sort controls on listing pages that can generate a combinatorial explosion of crawlable URLs from a small product set.
Duplicate content
Substantively identical or near-identical content available at more than one URL, within a site or across sites.
Crawl budget
The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.
robots.txt
A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.