Cloudflare Pay Per Crawl
Cloudflare launched a private beta letting publishers charge AI crawlers per request using the HTTP 402 Payment Required status code.
What changed
Pay Per Crawl gives site owners three options per crawler — allow free, charge a flat per-request price, or block — enforced at the edge using HTTP 402 responses and cryptographically signed crawler identity. Crawlers can either discover pricing reactively or declare a maximum price up front via request headers. It is the first infrastructure-level attempt to create a machine-readable market for content access rather than relying on bilateral licensing deals. The feature launched in private beta.
Timing
Rollout began 1 July 2025. No completion date was announced. Rankings typically remain unstable for the whole rollout window, so measurements taken inside it are not a reliable read on the outcome.