Reference library
Documentation
First-party documentation from search engines and platforms. These pages are authoritative by definition, when a blog post and the documentation disagree, the documentation wins.
60 references · 12 canonical · 59 free · Intermediate / Beginner / Advanced / All
| Title | Author / publisher | Year | Level | Cost |
|---|---|---|---|---|
| AI Features and Your Website ↗Canon Google's documentation on how AI Overviews and AI Mode source content, the eligibility requirements, and the preview controls site owners can use to limit inclusion. developers.google.com | Google Search Central | , | Intermediate | Free |
| Does Anthropic Crawl Data from the Web, and How Can Site Owners Block the Crawler? ↗Canon Anthropic's official help article describing ClaudeBot, Claude-User and Claude-SearchBot, their robots.txt tokens and the crawling practices Anthropic follows. support.claude.com | Anthropic | , | Intermediate | Free |
| Google Crawler (User Agent) Overview ↗Canon Authoritative list of Google's crawlers, fetchers and user-triggered agents, including Googlebot, Google-Extended (AI training and grounding opt-out), GoogleOther and their robots.txt tokens and IP ranges. developers.google.com | , | Intermediate | Free | |
| Google Search Central Blog ↗Canon Google's official channel for announcing algorithm updates, documentation changes, new structured data types and crawler behaviour changes. The primary-source record of Google Search announcements since 2005. developers.google.com | Google Search Central | , | All | Free |
| Google Search Console Help Center ↗Canon Official product help for Search Console, covering property types, the Performance report data model, Index Coverage states, URL Inspection and manual actions. support.google.com | , | All | Free | |
| Localized Versions of Your Pages (hreflang) ↗Canon Google's canonical reference for hreflang annotations across HTML, HTTP headers and sitemaps, including x-default and common implementation errors. developers.google.com | Google Search Central | , | Advanced | Free |
| Overview of OpenAI Crawlers ↗Canon OpenAI's official documentation of GPTBot (model training), OAI-SearchBot (search index) and ChatGPT-User (user-initiated fetches), with robots.txt tokens and published IP ranges. developers.openai.com | OpenAI | , | Intermediate | Free |
| SEO Starter Guide: The Basics ↗Canon Google's own introductory guide to search engine optimization, covering how Google finds and understands pages, how to influence appearance in results and which practices are unnecessary. developers.google.com | Google Search Central | , | Beginner | Free |
| Structured Data Markup that Google Search Supports ↗Canon The search gallery listing every structured data type that can produce a rich result in Google Search, with required and recommended properties for each. developers.google.com | Google Search Central | , | Intermediate | Free |
| The Beginner's Guide to SEO ↗Canon One of the longest-running free introductory SEO guides on the web, covering crawling and indexing, keyword research, on-page and technical optimisation, link building and measurement. moz.com | Moz | , | Beginner | Free |
| Understand JavaScript SEO Basics ↗Canon Describes Google's three-stage crawl, render and index pipeline for JavaScript applications and the practices that make client-rendered content indexable. developers.google.com | Google Search Central | , | Advanced | Free |
| Web Vitals ↗Canon Canonical definition of the Web Vitals initiative and the Core Web Vitals set, with the current thresholds and the field-versus-lab measurement model. web.dev | Google Chrome team | , | Intermediate | Free |
| About Applebot ↗ Apple's support article covering Applebot and Applebot-Extended, the user agents behind Siri, Spotlight Suggestions and Apple Intelligence training data, and the robots.txt controls for each. support.apple.com | Apple | , | Intermediate | Free |
| Ahrefs: Link Building for SEO: The Beginner's Guide ↗ Free guide to link acquisition tactics, prospecting, outreach and link evaluation, including which link types Ahrefs' own studies find correlate with rankings. ahrefs.com | Ahrefs | , | Intermediate | Free |
| Ahrefs: SEO: The Complete Guide for Beginners ↗ Ahrefs' free long-form SEO guide covering keyword research, on-page, link building, technical SEO and measurement, with worked examples from its own data. ahrefs.com | Ahrefs | , | Beginner | Free |
| Backlinko SEO Marketing Hub ↗ Free reference hub of SEO guides, definitions and data-backed studies, known for large-sample analyses of search results and CTR. backlinko.com | Backlinko | , | Beginner | Free |
| Baidu Search Resource Platform Documentation ↗ Baidu's webmaster platform documentation and announcements, covering submission APIs, crawling behaviour and the algorithm updates Baidu publishes for Chinese-language sites. ziyuan.baidu.com | Baidu | , | Advanced | Free |
| Bing Webmaster Tools Help Center ↗ Product documentation for Bing Webmaster Tools, including site verification, URL submission, crawl control, the Site Explorer and the IndexNow integration. bing.com | Microsoft Bing | , | All | Free |
| Block Search Indexing with noindex ↗ How to use the noindex rule via meta tag or X-Robots-Tag header, and why disallowing a URL in robots.txt prevents the rule from being seen. developers.google.com | Google Search Central | , | Beginner | Free |
| Brave Search Crawler Documentation ↗ Brave's help page on its independent search index crawler. Notes that the crawler does not advertise a differentiated user agent, respects Googlebot blocks, and that delisting requires a robots noindex directive rather than robots.txt. search.brave.com | Brave | , | Intermediate | Free |
| CCBot Crawler Documentation ↗ Common Crawl's page describing CCBot, the crawler that builds the open web corpus widely used to pretrain large language models, and how to allow or block it in robots.txt. commoncrawl.org | Common Crawl | , | Intermediate | Free |
| Chrome UX Report (CrUX) Documentation ↗ Documentation for the CrUX dataset of real-user Core Web Vitals from opted-in Chrome users, covering the API, BigQuery tables, the CrUX History API and dashboard tooling. developer.chrome.com | Google Chrome team | , | Advanced | Free |
| Cloudflare Bot Solutions Documentation ↗ Cloudflare's documentation on bot classification, verified bots and the controls network operators use to allow, challenge or block AI crawlers at the edge. developers.cloudflare.com | Cloudflare | , | Advanced | Freemium |
| Crawl Budget Management for Large Sites ↗ Google's explanation of crawl capacity limit and crawl demand, and the practical levers large sites have to influence how much Googlebot crawls. developers.google.com | , | Advanced | Free | |
| Create a News Sitemap ↗ Requirements for Google News sitemaps, including the news-specific XML namespace, the two-day article window and publication metadata. developers.google.com | Google Search Central | , | Intermediate | Free |
| Cumulative Layout Shift (CLS) ↗ Definition of CLS, the visual stability Core Web Vital, covering session windows, expected versus unexpected shifts and the layout shift score calculation. web.dev | Google Chrome team | , | Intermediate | Free |
| Debug Google Search Traffic Drops ↗ Google's structured diagnostic for Search traffic declines, distinguishing technical issues, manual actions, algorithmic changes, seasonality and reporting glitches. developers.google.com | Google Search Central | , | Intermediate | Free |
| Google Crawling and Indexing Documentation ↗ Hub of Google's technical documentation on how Googlebot discovers, crawls and indexes pages, covering robots.txt, sitemaps, canonicalization, redirects, mobile sites and JavaScript. developers.google.com | Google Search Central | , | Intermediate | Free |
| Google Search Status Dashboard: Ranking Updates History ↗ Google's authoritative log of confirmed ranking updates with official start and completion timestamps, used to align traffic changes with named algorithm rollouts. status.search.google.com | , | Intermediate | Free | |
| Google's Common Crawlers ↗ Detail page for Google's common crawlers, listing user-agent strings, robots.txt tokens and reverse-DNS verification for Googlebot variants. developers.google.com | , | Intermediate | Free | |
| How HTTP Status Codes Affect Google's Crawlers ↗ Maps each HTTP status code, network error and DNS failure to the behaviour Google's crawlers and indexing systems apply. developers.google.com | , | Advanced | Free | |
| How to Specify a Canonical with rel=canonical and Other Methods ↗ Explains how Google consolidates duplicate URLs, which canonicalization signals it uses and how to troubleshoot unexpected canonical selection. developers.google.com | Google Search Central | , | Intermediate | Free |
| How To Use Search Console ↗ Task-based guide to Search Console for common roles, linking to property verification, performance reporting, indexing reports and URL inspection. developers.google.com | Google Search Central | , | Beginner | Free |
| How to Write Meta Descriptions and Control Snippets ↗ Google's guidance on snippet generation, meta description quality and the directives that limit snippet length or suppress snippets entirely. developers.google.com | Google Search Central | , | Beginner | Free |
| Influencing Title Links in Google Search ↗ Explains how Google generates the title link shown in results, which sources it draws on and why the title element is sometimes rewritten. developers.google.com | Google Search Central | , | Beginner | Free |
| Interaction to Next Paint (INP) ↗ Definition of INP, the responsiveness metric that replaced First Input Delay as a Core Web Vital, including input delay, processing time and presentation delay components. web.dev | Google Chrome team | , | Intermediate | Free |
| Intro to How Structured Data Markup Works ↗ Google's primer on structured data formats (JSON-LD, microdata, RDFa), placement, testing and the relationship between markup and rich result eligibility. developers.google.com | Google Search Central | , | Beginner | Free |
| Intro to Product Structured Data on Google ↗ Reference for Product, Offer, AggregateRating and merchant listing markup used for product rich results and Google Shopping surfaces. developers.google.com | Google Search Central | , | Intermediate | Free |
| JSON-LD Project Site ↗ Community home for JSON-LD with the playground, tooling list and specification links. Useful for debugging structured data before deploying it. json-ld.org | JSON-LD Community | , | Intermediate | Free |
| Largest Contentful Paint (LCP) ↗ Definition and measurement methodology for LCP, including which elements qualify, the good/needs-improvement/poor thresholds and the sub-part breakdown for optimisation. web.dev | Google Chrome team | , | Intermediate | Free |
| MDN: HTTP Response Status Codes ↗ Practical reference for every HTTP status code with browser and crawler implications. The most commonly consulted working reference when auditing redirect and error handling. developer.mozilla.org | Mozilla | , | Intermediate | Free |
| MDN: The link Element ↗ Reference for the HTML link element and its rel values, including canonical, alternate, preload and preconnect, with attribute-level browser support notes. developer.mozilla.org | Mozilla | , | Beginner | Free |
| Meta Tags and Attributes that Google Supports ↗ Complete list of meta tags and HTML attributes Google Search recognises, including google-site-verification, notranslate, nositelinkssearchbox and data-nosnippet. developers.google.com | Google Search Central | , | Intermediate | Free |
| Meta Web Crawlers Documentation ↗ Meta's developer documentation for its crawlers, including facebookexternalhit for link previews and meta-externalagent, and the robots.txt directives that control them. developers.facebook.com | Meta | , | Intermediate | Free |
| Moz: Link Building: The Free SEO Beginner's Guide ↗ Moz's free chaptered guide to link building strategy, tactics, outreach and measurement, including guidance on manual penalties and disavowal. moz.com | Moz | , | Intermediate | Free |
| Perplexity Crawlers Documentation ↗ Perplexity's documentation of PerplexityBot and Perplexity-User, explaining which agent indexes content for its answer engine and which fetches pages on behalf of a user request. docs.perplexity.ai | Perplexity AI | , | Intermediate | Free |
| Redirects and Google Search ↗ Covers permanent and temporary server-side redirects, meta refresh, JavaScript redirects and how each is interpreted for canonicalization and site moves. developers.google.com | Google Search Central | , | Intermediate | Free |
| Remove Your Site Info from Google ↗ Decision guide for temporary URL removals, permanent removals, outdated content removal and refresh requests in Search Console. developers.google.com | Google Search Central | , | Intermediate | Free |
| Robots Meta Tags Specifications ↗ Specification of the robots meta tag and X-Robots-Tag HTTP header directives Google supports, including noindex, nofollow, nosnippet, max-snippet, max-image-preview and noai-adjacent controls. developers.google.com | Google Search Central | , | Intermediate | Free |
| Robots.txt Introduction and Guide ↗ Google's reference for what robots.txt can and cannot do, syntax supported by Googlebot, and the interaction between disallow rules and indexing. developers.google.com | Google Search Central | , | Intermediate | Free |
| Schema.org Getting Started Guide ↗ Introductory walkthrough of schema.org markup covering item types, properties, nesting and the differences between microdata, RDFa and JSON-LD encodings. schema.org | Schema.org | , | Beginner | Free |
| Search Engine Journal: What Is SEO? An Introduction to SEO Basics ↗ Search Engine Journal's free introductory SEO guide and topic hub, updated as the discipline changes and linked to its ebook library. searchenginejournal.com | Search Engine Journal | , | Beginner | Free |
| Search Engine Land: What Is SEO? ↗ Search Engine Land's evergreen explainer and guide hub defining SEO, its disciplines and how the trade press categorises ranking factors and search updates. searchengineland.com | Search Engine Land | , | Beginner | Free |
| SEO Best Practices for Ecommerce Sites ↗ Google's specialty documentation for online stores, covering URL structure for facets, pagination, product data, out-of-stock handling and merchant listings. developers.google.com | Google Search Central | , | Intermediate | Free |
| Site Names in Google Search ↗ Requirements for influencing the site name displayed above results, including WebSite structured data and other supported signals. developers.google.com | Google Search Central | , | Beginner | Free |
| Understanding Google Page Experience ↗ Google's current position on page experience signals, including Core Web Vitals, HTTPS and intrusive interstitials, and how they relate to ranking. developers.google.com | Google Search Central | , | Intermediate | Free |
| Using robots.txt (Yandex) ↗ Yandex's robots.txt reference, including directives such as Clean-param and Host that are specific to the Yandex robot. yandex.com | Yandex | , | Intermediate | Free |
| What Is a Sitemap (Google Search Central) ↗ Google's guidance on when XML sitemaps help, the supported formats, sitemap index files and submission methods. developers.google.com | Google Search Central | , | Beginner | Free |
| Where Do DuckDuckGo Search Results Come From? ↗ DuckDuckGo's explanation of its result sources, including its own DuckDuckBot crawler, Bing, and other partners. Clarifies why there is no separate DuckDuckGo submission process. duckduckgo.com | DuckDuckGo | , | Beginner | Free |
| Yandex Webmaster Help for Site Owners ↗ Yandex's official English-language help for webmasters, covering site indexing, Yandex Webmaster tools, quality requirements and the behaviour of Yandex's robot. yandex.com | Yandex | , | Intermediate | Free |