Cloudflare Pay Per Crawl
Cloudflare launched a private beta letting publishers charge AI crawlers per request using the HTTP 402 Payment Required status code.
What changed
Pay Per Crawl gives site owners three options per crawler (allow free, charge a flat per-request price, or block) enforced at the edge using HTTP 402 responses and cryptographically signed crawler identity. Crawlers can either discover pricing reactively or declare a maximum price up front via request headers. It is the first infrastructure-level attempt to create a machine-readable market for content access rather than relying on bilateral licensing deals. The feature launched in private beta.
Timing
Rollout began 1 July 2025. No completion date was announced. Rankings typically remain unstable for the whole rollout window, so measurements taken inside it are not a reliable read on the outcome.