Cloudflare will begin blocking multi-purpose crawlers on ad-supported webpages for all new and free-tier customers starting September 15, 2024. This change in default settings for its bot management protocol applies to websites with advertisements. Approximately one-fifth of the world's websites are hosted by Cloudflare.

Cloudflare's chief strategy officer, Stephanie Cohen, stated the company's objective. "We've been clear about what we want. We want a technical solution that allows you to be discoverable without having to give your content away for free," Cohen said. Google utilizes a single crawler for both indexing websites and training its artificial intelligence models.

The creator network Beehiiv announced a partnership with Cloudflare, enabling Beehiiv's network of creators to block the Google crawler. A Google spokesperson stated that the company provides web publishers with controls to manage their content, including Google-Extended for AI training and a new Search Console control for generative AI Search features. "We provide web publishers with clear, granular controls to manage their content, including Google-Extended for AI training and a new Search Console control we are testing for generative AI Search features, neither of which impact traditional Search visibility. We are committed to designing AI experiences that highlight the web, drive valuable traffic to publishers, and provide the insights they need to succeed," the spokesperson said. Google Extended allows publishers to opt out of AI training while maintaining visibility in Search.