Index bloat
A condition where a site has far more URLs indexed or crawlable than it has genuinely useful pages, diluting crawl attention and…
Filter and sort controls on listing pages that can generate a combinatorial explosion of crawlable URLs from a small product set.
Each filter combination typically produces a distinct URL, so a handful of facets can yield millions of near-duplicate pages. The standard approach is to decide which facet combinations have genuine search demand and make those indexable with unique content and internal links, while blocking or noindexing the rest and stripping ordering, view and session parameters. Google's e-commerce guidance recommends using URL patterns crawlers can recognise and, where a facet space is truly infinite, disallowing it in robots.txt.
`/running-shoes/mens/waterproof/` is indexable, while `/running-shoes?sort=price&view=grid&page=17` is disallowed.
A condition where a site has far more URLs indexed or crawlable than it has genuinely useful pages, diluting crawl attention and…
The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.
Key-value pairs after a question mark in a URL; they frequently create duplicate content and crawl waste when they do not change page…
Splitting a long list or article across sequential URLs; Google stopped using rel=next/prev for indexing in 2019 and treats each page…
A page listing multiple products, such as a category, collection or filtered view; usually the highest-value organic template on a retail…