Cloudflare and GoDaddy Launch Tools to Block and Monetize AI Crawlers

Cloudflare announced a series of measures to protect digital content from AI web crawlers, beginning with the default blocking of known scrapers for new domain owners. The internet architecture provider introduced a Pay Per Crawl program for a select group of publishers and creators, enabling them to charge AI companies for access to their content. This initiative addresses a decline in web traffic as users increasingly rely on AI chatbots over traditional search engines.

To expand these capabilities, Cloudflare partnered with GoDaddy to integrate AI Crawl Control into the hosting platform. This allows website owners to block, permit, or demand payment from AI agents. The companies are also promoting new identity standards to verify automated systems, including the Agent Name Service, an open standard introduced by GoDaddy using Domain Name System and Public Key Infrastructure, and Cloudflare's Web Bot Auth, which employs cryptography and Signature Agent Cards to verify bot identities and purposes.