Pricing the Open Web

When Microsoft launched its Publisher Content Marketplace in February, the company put the problem plainly.

That model does not translate cleanly to an AI-first world, where answers are increasingly delivered in a conversation. — Microsoft

The marketplace was Microsoft's answer: a platform where publishers set their own terms and pricing, replacing fragmented one-off deals with something systematic. The move was notable not for the sentiment but for the machinery. Microsoft had built something to price the gap. Within days, Amazon was exploring its own centralized content marketplace, pitching publishers on commission-based revenue as what one report called a "legally safer alternative to scraping." In July, Cloudflare replaced its Pay Per Crawl model with Pay Per Use and began urging bot operators to split their crawlers into three separately priceable categories: Search, Agent, and Training.

Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training. — Matthew Prince

Earlier in the year, the BBC, the Financial Times, Sky News, The Telegraph, and The Guardian formed the SPUR coalition to establish shared technical standards and machine-readable usage signals for licensing journalistic content to AI, with the stated goal of shaping "a market that rewards original reporting." ^[1] In June, the UK Competition and Markets Authority imposed legally enforceable conduct requirements on Google, mandating publisher opt-out tools, attribution links, and engagement-metric sharing. Google's attribution links were a regulatory compliance measure, not a voluntary concession. ^[2] Lined up, these moves form a pattern that no single story captures. Across platforms, regulators, and publishers, multiple actors are independently building infrastructure to make content friction a priced, routable thing. The pieces differ in origin. One is a regulatory mandate; the rest are market moves. They all point in the same direction. None of this signals retreat. Big Tech firms spent $234 billion in AI capital expenditures in the first half of 2026 alone, with Meta, Amazon, and Nvidia each raising $25 billion in bonds to fund the spree. ^[3] OpenAI raised $122 billion at an $852 billion valuation in March, targeting $1.4 trillion in chips and data center spending. ^[4] Global AI infrastructure spending is projected to reach $1.7 trillion by 2030. ^[5] The friction infrastructure and the spending spree are simultaneous, not sequential. The industry has not chosen between growth and friction. It has made friction a line item: something to build systems around, price, and route through, while expanding aggressively on those new terms. The European Publishers Council captured what is at stake in its antitrust complaint against Google earlier this year. Chairman Christian Van Thillo made the point in existential terms.

No amount of money can restore lost audiences, weakened brand relationships or eroded reader trust once publishers are disintermediated. — Christian Van Thillo

Van Thillo's point was structural, not transactional: a claim about what happens when AI search disintermediates publishers entirely. Cloudflare's bot-splitting taxonomy of Search, Agent, and Training is the other side of the same coin: market machinery that assumes the friction is permanent and needs to be routed, not resolved. Both sides are building infrastructure. Neither is negotiating a truce.