IETF Develops Web Standards to Distinguish AI Bots from Search

The Internet Engineering Task Force is developing new web standards designed to help website owners differentiate between traditional search engines and generative AI answer engines. These standards would enable publishers to block AI bot crawlers that scrape data for model training and direct answers, which often remove the need for users to visit the source website, while still permitting search bots that provide referral traffic.

Major technology companies, including Google, Microsoft, OpenAI, and Amazon, are lobbying to narrow or postpone these standards. These firms argue that AI and search have become too intertwined to separate effectively. OpenAI specifically warned that the guidelines could be used by regulators to limit data access. In response to industry pressure, a recent version of the standards removed the AI Use category, though the core definition of search as an application that directs users to the original content location remains.

Content creators and publishers support the initiative, arguing that AI summaries substitute for original content and threaten existing revenue models. A Mozilla engineer is currently co-authoring the working document to ensure AI summaries remain distinguished from search applications. The development process is expected to conclude by the end of 2025.