Cloudflare introduces feature to block AI training crawlers while allowing search engines

Cloudflare has introduced a new feature called 'Disallow AI Training,' enabling website administrators to block AI training crawlers while still allowing search engines to crawl their sites. Major tech companies including Apple, Google, and Microsoft have committed to adhering to this setting. Once activated, the feature will permit mixed crawlers for search and AI training, provided they are marked as 'Accountable.' Other AI training crawlers will remain blocked, and Cloudflare warned that blocking these crawlers could impact a site's search rankings. Most customers are unaffected, as websites previously set to 'Block' or 'Block on pages with ads' will automatically transition to the new setting. Additionally, Google is set to release an extended URL-level transparency tool in the coming weeks, while Apple is developing its own URL-level tool. Microsoft plans to support rejecting AI training through robots.txt in early 2027.