Cache-Control
The HTTP header governing how browsers, CDNs and proxies store and revalidate a response, using directives such as max-age, s-maxage and…
TechnicalIf-None-MatchIf-Modified-Since
ETag and conditional requests
A validator token identifying a specific version of a resource, allowing clients and crawlers to revalidate with a cheap 304 Not Modified response.
In full
When a crawler sends `If-None-Match` with a stored ETag, or `If-Modified-Since` with a stored date, an unchanged resource can be answered with a 304 carrying no body. Google supports conditional requests, so correct ETag and Last-Modified handling saves bandwidth on both sides and effectively stretches crawl capacity on large sites. ETags must change when content changes, or crawlers will keep serving stale copies from cache.
Example
A 500,000-URL catalogue implements ETags and sees 60% of Googlebot fetches answered with 304, freeing capacity for new products.
Related terms
Crawl budget
The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.
HTTP status codes
Numeric codes in an HTTP response that tell crawlers how to treat a URL: 200 index it, 3xx follow the redirect, 404/410 drop it, 5xx come…
Content Delivery Network
A distributed network of edge servers that caches and serves content close to users, reducing latency and offloading the origin.
Log file analysis
Examining raw server access logs to see exactly which URLs crawlers fetched, how often, with what status codes and response times.