SEO Atlas
Crawling IndexingSpidering

Crawling

The process by which a search engine discovers URLs and downloads their content using automated software, before any indexing or ranking happens.

In full

A crawler starts from known URLs — sitemaps, previously crawled pages, links — and follows references to discover more. Each fetch is subject to robots.txt rules, host load limits and the crawler's own scheduling priorities. Crawling only obtains bytes; whether those bytes are stored and made retrievable is a separate decision made during indexing. A page that is never crawled can still be indexed URL-only if it is linked to, but Google will have no content to describe it with.

Example

A news site publishes a new article, pings its sitemap, and Googlebot fetches the URL within minutes; the article is crawled but does not appear in search until it is also indexed.

Related terms

Indexing

Storing and organising a crawled page in a search engine's index so it becomes eligible to be retrieved and ranked for queries.

Googlebot

Google's main web crawler, which fetches pages for Google Search using two user agents: Googlebot Smartphone and Googlebot Desktop.

robots.txt

A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.

Crawl budget

The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.

XML sitemap

An XML file listing URLs a site wants crawled, optionally with lastmod dates, used as a discovery aid by Google, Bing and other engines.

Where to read more

Tools that touch this

advertoolsBotifyCrawleeJetOctopusKatanaLibreCrawl Technical SEO Audit MCP