SEO Atlas
Crawling IndexingGoogle crawler

Googlebot

Google's main web crawler, which fetches pages for Google Search using two user agents: Googlebot Smartphone and Googlebot Desktop.

In full

Googlebot crawls from Google-owned IP ranges that are published as reverse-DNS-verifiable JSON files, so site owners can distinguish it from spoofers. Since mobile-first indexing, the primary crawler for nearly all sites is Googlebot Smartphone. Googlebot obeys robots.txt, supports HTTP/2 where the server offers it, and fetches at most a few megabytes of HTML per URL. It is distinct from Google's other agents such as GoogleOther, Google-Extended, AdsBot and Google-InspectionTool.

Example

Server logs show `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 ... Googlebot/2.1` hitting a category page 40 times a day.

Related terms

Crawling

The process by which a search engine discovers URLs and downloads their content using automated software, before any indexing or ranking…

Mobile-first indexing

Google's practice of crawling, indexing and ranking based on the mobile version of a page; completed for all sites by mid-2024.

robots.txt

A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.

Google-Extended

A robots.txt control token that governs whether a site's content can be used to improve Gemini models and grounded Gemini API responses…

Log file analysis

Examining raw server access logs to see exactly which URLs crawlers fetched, how often, with what status codes and response times.

Where to read more

Tools that touch this

Google robots.txt Parser