Cloudflare Launches AI Crawler Controls and Publisher Payment Model

Cloudflare announced a series of tools and policy changes aimed at decoupling traditional search indexing from AI data extraction to create a sustainable economic model for publishers. Starting September 15, 2026, the company will implement new default settings for new websites and existing free customers. These defaults will allow traditional search crawling but block AI training and agent use on advertising-supported pages. Because many crawlers from companies like Google, Microsoft, and Apple perform both functions, they will be blocked on ad-supported pages if a site restricts training.

As part of its Agentic Internet vision, the company is replacing its Pay Per Crawl experiment with a Pay Per Use model. This system enables publishers to receive compensation when their content appears in AI results or when agents access premium information. Cloudflare has launched this program with initial partners Ceramic.ai and You.com. To support these efforts, the company is debuting an Attribution Business Insights dashboard to track bot consumption and testing new signals in robots.txt to reduce crawl waste.

Cloudflare is urging bot operators to separate their crawlers into three distinct behavioral categories: Search, Agent, and Training. This classification aims to provide transparency, allowing website owners to remain discoverable via search engines while preventing the unauthorized use of their data for model training. The company also introduced BotBase for Enterprise users to further manage these interactions.