Crawling IndexingSitemap

XML sitemap

An XML file listing URLs a site wants crawled, optionally with lastmod dates, used as a discovery aid by Google, Bing and other engines.

In full

A single sitemap file is limited to 50,000 URLs and 50 MB uncompressed. Google uses `<lastmod>` when it is consistently accurate and ignores `<priority>` and `<changefreq>` entirely. Sitemaps should contain only canonical, indexable, 200-status URLs; padding them with redirects or noindexed pages erodes trust in the file. They can be submitted in Search Console, referenced from robots.txt, or pinged via IndexNow-style protocols.

Example

A 300,000-URL retailer ships 7 sitemap files referenced from one sitemap index, each segmented by product category so indexing gaps are easy to spot.

Related terms

Sitemap index file

An XML file that lists other sitemap files, letting a site exceed the 50,000-URL per-file limit and organise sitemaps by section.

IndexNow

An open protocol for pushing URL change notifications to participating search engines, adopted by Microsoft Bing, Yandex, Seznam and Naver.

Crawling

The process by which a search engine discovers URLs and downloads their content using automated software, before any indexing or ranking…

Google News sitemap

A specialised sitemap containing only articles published in the last two days, using the news namespace to speed discovery for Google News…

Image and video sitemap extensions

Namespace extensions to XML sitemaps that declare images and videos associated with a page so they can be discovered for Images and Video…

Where to read more

Tools that touch this