Crawling Indexing

Canonicalization

Duplicate consolidation

The process by which a search engine groups duplicate or near-duplicate URLs into a cluster and picks one representative URL to index and rank.

Cloaking

Presenting different content or URLs to search engines than to users with the intent to manipulate rankings; a Google spam policy violation.

Crawl budget

The number of URLs a search engine is willing and able to crawl on a site in a given period, set by crawl capacity limit and crawl demand.

Crawl demand

How much a search engine wants to crawl a set of URLs, driven by their popularity, how often they change, and the site's perceived overall quality.

Crawl rate limit

Crawl capacity limit

The maximum simultaneous connections and fetch frequency a crawler will use for a host, adjusted automatically based on server response times and errors.

Crawl trap

Infinite space · Spider trap

A URL pattern that generates effectively unlimited unique URLs, causing crawlers to loop indefinitely and exhaust crawl budget.

Crawled — currently not indexed

A Search Console status meaning Google fetched a URL but chose not to index it, usually a quality, duplication or value judgement.

Crawling

Spidering

The process by which a search engine discovers URLs and downloads their content using automated software, before any indexing or ranking happens.

data-nosnippet

An HTML attribute that excludes a specific span, div or section of a page from being used in Google's snippets while leaving the rest eligible.

Discovered — currently not indexed

A Search Console status meaning Google knows a URL exists but has not crawled it yet, typically because of crawl capacity or low crawl demand.

DOM (Document Object Model)

Rendered HTML

The in-memory tree representation of a document that the browser builds from HTML and mutates with JavaScript; the rendered DOM is what search engines index.

Duplicate content

Substantively identical or near-identical content available at more than one URL, within a site or across sites.

Dynamic rendering

Serving a server-rendered HTML version to crawlers and a client-rendered version to users; Google now describes it as a deprecated workaround.

Faceted navigation

Filtered navigation · Facets

Filter and sort controls on listing pages that can generate a combinatorial explosion of crawlable URLs from a small product set.

Google News sitemap

News sitemap

A specialised sitemap containing only articles published in the last two days, using the news namespace to speed discovery for Google News and Top stories.

Googlebot

Google crawler

Google's main web crawler, which fetches pages for Google Search using two user agents: Googlebot Smartphone and Googlebot Desktop.

Image and video sitemap extensions

Image sitemap · Video sitemap

Namespace extensions to XML sitemaps that declare images and videos associated with a page so they can be discovered for Images and Video search.

Index bloat

A condition where a site has far more URLs indexed or crawlable than it has genuinely useful pages, diluting crawl attention and site-level quality signals.

Indexing

Index selection

Storing and organising a crawled page in a search engine's index so it becomes eligible to be retrieved and ranked for queries.

IndexNow

An open protocol for pushing URL change notifications to participating search engines, adopted by Microsoft Bing, Yandex, Seznam and Naver.

Infinite scroll

Endless scrolling

A UI pattern that appends more items as the user scrolls; without backing paginated URLs the additional items are usually invisible to crawlers.

JavaScript rendering

JavaScript SEO

The execution of a page's client-side JavaScript by a search engine so that content and links generated at runtime become indexable.

Log file analysis

Server log analysis

Examining raw server access logs to see exactly which URLs crawlers fetched, how often, with what status codes and response times.

noindex

A robots directive, delivered by meta tag or X-Robots-Tag header, that instructs search engines to drop a page from their index.

nosnippet / max-snippet

Snippet controls

Robots directives controlling whether and how much of a page's text a search engine may show as a snippet or use in generated summaries.

Orphan page

A page with no internal links pointing to it, reachable only via sitemap, external link or direct URL.

Pagination

Paginated series

Splitting a long list or article across sequential URLs; Google stopped using rel=next/prev for indexing in 2019 and treats each page independently.

Prerendering

Static prerendering

Generating and caching a fully rendered HTML snapshot of a JavaScript page at build time or on request, so crawlers receive complete markup.

rel="canonical"

Canonical tag · Canonical link element

A link annotation (in the head or an HTTP header) that tells search engines which URL you consider the preferred version of duplicate content.

rel="nofollow"

nofollow

A link attribute telling search engines not to pass ranking credit through a link; since 2019 Google treats it as a hint rather than a directive.

rel="sponsored"

Sponsored link attribute

A link attribute introduced by Google in 2019 to identify links created as part of advertising, sponsorships or other compensation agreements.

rel="ugc"

User-generated content link attribute

A link attribute marking links that originate in user-generated content such as comments, forum posts and profile fields.

Removals tool

Temporary removals

A Search Console tool that temporarily hides a URL from Google results for about six months while a permanent removal method is put in place.

Render queue

Second wave of indexing · Web Rendering Service

The deferred queue in which Google executes a page's JavaScript with a headless Chromium instance after the raw HTML has already been crawled.

Robots meta tag

meta robots

A `<meta name="robots">` element in the HTML head carrying per-page indexing and serving directives such as noindex, nofollow, nosnippet and max-image-preview.

robots.txt

Robots Exclusion Protocol · REP

A plain-text file at the root of a host that tells compliant crawlers which URL paths they may or may not fetch, standardised as RFC 9309.

Sitemap index file

Sitemap index

An XML file that lists other sitemap files, letting a site exceed the 50,000-URL per-file limit and organise sitemaps by section.

Soft 404

A URL that returns HTTP 200 while displaying content that means 'not found' or has no real content, causing search engines to treat it as an error.

URL Inspection tool

URL Inspection API

A Search Console feature showing Google's indexed state for a URL plus a live test with rendered HTML, screenshot, and page resource loading results.

URL parameters

Query strings

Key-value pairs after a question mark in a URL; they frequently create duplicate content and crawl waste when they do not change page content.

X-Robots-Tag

An HTTP response header that carries the same robots directives as the meta robots tag, usable for any file type including PDFs and images.

XML sitemap

Sitemap

An XML file listing URLs a site wants crawled, optionally with lastmod dates, used as a discovery aid by Google, Bing and other engines.