Open standards development organisation established in 1986 that produces the RFC series governing internet protocols. It standardised the Robots Exclusion Protocol as RFC 9309 in 2022 and hosts working groups on HTTP, TLS and machine-readable content preferences, all of which set the rules crawlers must follow.
RFC 9309 (Robots Exclusion Protocol); HTTP and TLS specifications.
Internet Engineering Task Force (IETF) is a standards body based in Global, founded in 1986. Membership is free.
6 other standards bodies, plus others based in Global.
Independent auditor of media and advertising data, verifying circulation, digital traffic and ad-supply data for publishers and buyers. Its audits are one of the few third-party verifications of publisher traffic claims, relevant when evaluating link and placement partners.
Coalition maintaining an open technical standard, Content Credentials, that records the origin and edit history of digital media. Provenance metadata is increasingly used by search and AI answer engines as a trust and authenticity signal for images and video.
Metadata standards organisation that arose from a workshop series in the 1990s and now operates as a project of ASIS&T. It maintains the Dublin Core Metadata Element Set, standardised as ISO 15836, which remains a foundational vocabulary for describing web resources and is reused inside Schema.org and library-sector SEO.
The single authoritative voice on advertising self-regulation in Europe, coordinating national self-regulatory organisations to keep advertising legal, decent, honest and truthful. Its cross-border complaint mechanism and guidance, including on generative-AI disclosure, apply to online and search advertising across European markets.
Non-profit standards consortium established in 2014 that develops the technical specifications underpinning digital advertising, including OpenRTB, ads.txt and privacy signalling. Its newer agentic and AI workstreams are defining how AI agents identify themselves to publishers, which overlaps directly with crawler management.
IETF working group chartered to standardise how site owners express preferences about the use of their content by AI systems, including a common vocabulary and a means of attaching those preferences to web content. It is the venue where the successor to ad-hoc AI crawler directives in robots.txt is being negotiated.