Cloudflare one-click AI crawler blocking
Cloudflare gave all customers, including free plans, a single toggle to block AI training crawlers at the network edge.
What changed
Cloudflare shipped a feature that blocks known AI crawlers regardless of robots.txt compliance, enforcing the choice at the network layer rather than relying on voluntary respect for directives. It also published bot-behaviour fingerprinting to catch crawlers that disguise themselves. Because Cloudflare fronts a very large share of the web, the change materially altered what AI companies could access. It was the first step toward the default-block and paid-access model Cloudflare introduced a year later.
Timing
Rollout began 3 July 2024. No completion date was announced. Rankings typically remain unstable for the whole rollout window, so measurements taken inside it are not a reliable read on the outcome.