Cloudflare blocks AI crawlers by default
Cloudflare made blocking AI crawlers the default for new domains, ending open-by-default access to a large share of the web.
What changed
Cloudflare announced that new domains on its network would block AI crawlers unless the site owner opts in, reversing the long-standing assumption that crawlers may access content unless told otherwise. It framed the change around the collapsing ratio of crawls to referral visits from AI platforms compared with traditional search. Because Cloudflare fronts a very large portion of web traffic, the policy shifted the practical default for AI data access across the open web.
Timing
Rollout began 1 July 2025. No completion date was announced. Rankings typically remain unstable for the whole rollout window, so measurements taken inside it are not a reliable read on the outcome.