Cloudflare Blocks AI Crawlers by Default and Launches Pay-Per-Crawl
Cloudflare now blocks AI crawlers by default for new customers and introduced a marketplace for website owners to charge AI companies for content access.
Cloudflare has updated its infrastructure to block AI crawlers by default for all new customers, reversing the industry standard of permitting bots unless explicitly blocked. To address the lack of traffic referrals from AI developers, the company launched Pay-Per-Crawl, a marketplace that enables website owners to charge AI companies a fee for every page crawled.
Under the new system, AI bots must cryptographically identify themselves and complete payment through Cloudflare before content is served. Matthew Prince, CEO of Cloudflare, stated that while AI companies may be reluctant to be the first to pay, the market is moving toward this model. He noted that the system could eventually help AI companies find the most valuable or freshest content to make crawling more productive.
Several major publishers, including Gannett, Condé Nast, The Atlantic, BuzzFeed, and Time, have already joined the platform to protect and monetize their work. The shift follows reports of a significant imbalance in crawl-to-referral ratios, with companies like OpenAI and Anthropic providing far fewer referrals to websites than traditional search engines such as Google.
