DocumentationIrregular

RFC 9309 — Robots Exclusion Protocol

IETF standards-track specification of robots.txt, formalising the Robots Exclusion Protocol in 2022 after decades as a de facto convention.

What it covers

IETF standards-track specification of robots.txt, formalising the Robots Exclusion Protocol in 2022 after decades as a de facto convention.

RFC 9309 — Robots Exclusion Protocol publishes irregular in English, from United States, and has been running since 2022. It is published by IETF. Content is freely accessible.

Topics

Notable voices

More like this

6 other documentations.

All documentation →
DocumentationIrregular

Baidu's official webmaster platform and documentation, the primary reference for indexing and ranking guidance in the Chinese search market.

ziyuan.baidu.com
DocumentationIrregular

Official Microsoft blog for Bing Webmaster Tools, covering IndexNow, Bing crawling and indexing behaviour, and Copilot-era search changes.

blogs.bing.com
DocumentationIrregular

Microsoft's official guidance for how Bing crawls, indexes and ranks content, including abuse policies and technical requirements.

bing.com
DocumentationWeekly

Official Chrome developer blog covering rendering, JavaScript, Chrome UX Report data and browser changes with direct implications for technical SEO.

developer.chrome.com
DocumentationIrregular

Technical SEO reference library built by ContentKing, now redirected into Conductor's Technical SEO Hub after the 2022 acquisition. Remains a widely cited source of technical SEO definitions.

conductor.com
DocumentationMonthly

Official Google blog for site owners and SEOs, announcing ranking system updates, spam policy changes, structured data support and crawling behaviour.

developers.google.com