Google Tests Web Bot Auth to Verify Crawler Identity

Google is testing Web Bot Auth, an experimental protocol designed to help website owners verify the identity of automated traffic. The system utilizes cryptographic credentials stored in JSON Web Key Set format at standardized /.well-known/ addresses to distinguish legitimate crawlers from rogue bots that spoof user-agent strings.

By incorporating a Signature-Agent header into HTTP requests, the protocol enables websites to validate a bot's identity without requiring manual security key exchanges. The system is based on the HTTP Message Signatures Directory standard and is currently applied to a limited subset of traffic, such as Google-Agent.

Because the protocol is in an experimental phase, the company advises developers to maintain existing verification methods. Google recommends that website owners continue relying on IP addresses, reverse DNS, and user-agent strings to prevent the accidental blocking of legitimate traffic during the gradual rollout.