Cloudflare Launches Tool to Block AI Training Crawlers

Cloudflare CEO Matthew Prince announced that automated bot and AI agent traffic surpassed human web traffic in May, accounting for nearly 60% of all website requests. To combat the unauthorized ingestion of content by AI crawlers, Cloudflare launched a Disallow AI Training setting.

This tool enables website owners to block AI companies from using their content for model training while ensuring the sites remain indexed for traditional search engines. Apple Inc., Google, and Microsoft have already agreed to the deal.

Prince stated that AI companies must pay content creators for the data that fuels their systems to maintain a sustainable media model. He warned that without a mechanism to fund content creators and infrastructure, the future of the internet is uncertain.