Web Bot Auth — Definition and What It Changes for Stores
Web Bot Auth is a way for a bot to prove who it is: it signs each HTTP request with a private key, and the site verifies that signature against a public key the bot publishes at a well-known URL. It replaces a claim with a proof. A user-agent string says “I am GPTBot” and anyone can type that; a valid signature can only be produced by the party holding the key. The work is being standardised at the IETF in the Web Bot Auth working group and builds on HTTP Message Signatures (RFC 9421).
Cryptographic identity for crawlers and AI agents — the bot signs, the origin verifies, and nobody has to trust a header any more.
How it works
- The bot operator generates a key pair, usually Ed25519, and publishes the public half as a JWK Set at a well-known key directory URL.
- Each outgoing request carries a signature over selected request components, using HTTP Message Signatures.
- The request also carries a
Signature-Agentheader naming where the key directory lives, so the origin can find the key in band. - The origin fetches the directory, verifies the signature, and now knows which operator sent the request — regardless of what the user-agent string says.
An optional Signature Agent Card — a JSON document at the directory — lets the operator describe itself: identity, purpose, expected request rate, and keys.
Where the standard stands
| Document | What it covers | Status |
|---|---|---|
draft-meunier-webbotauth-httpsig-protocol | The signing protocol, Signature-Agent header, JWKS directory and its well-known URI | Internet-Draft, standards track (rev 01, August 2026) |
draft-meunier-webbotauth-registry | The Signature Agent Card and an IANA registry for its members | Internet-Draft, informational (rev 03, June 2026) |
draft-meunier-web-bot-auth-architecture | The overall architecture | Internet-Draft |
None of these is an RFC yet. Authors come from Cloudflare, Google and Amazon, which is why implementations exist ahead of publication.
Who signs today
Google’s documentation states that not all of its user agents use Web Bot Auth, that it is not yet signing every request from the agents that do, and that origins should keep using IP ranges, reverse DNS and user-agent strings alongside the signature. That is a vendor telling you not to enforce on the signature alone. Cloudflare shipped verification into its Verified Bots programme in July 2025; AWS added support in AWS WAF in November 2025, initially for CloudFront distributions only.
Read that as a floor, not a finished system. A signed request is strong evidence. An unsigned request is not evidence of anything — it may come from an operator that has not rolled out signing yet.
What it does not solve
Identity is not intent. A signature proves which operator sent the request; it says nothing about whether that operator will respect your rate limits, your licence terms or your prices. And the traffic most stores actually struggle with — scrapers, scalper toolkits, resellers of a competitor’s free tier — will simply not sign anything.
So the useful shape is an allowlist, not a blocklist: treat signed, verified traffic from operators you want as first-class, and keep the rest under the controls you already run.
Why an ecommerce store should care
- Fewer false blocks. A verified AI search crawler can be admitted through a WAF rule that would otherwise catch it as suspicious traffic — which is how stores lose AI visibility without ever deciding to.
- Rate limits that mean something. Per-operator limits only work if operator identity is real.
- The same key material as UCP. A UCP business profile publishes its public keys as a JWK Set in
keys[]. If you already publish keys for signed UCP responses, you already run the piece Web Bot Auth needs on the other side of the conversation.
What to do now on Magento 2
Signature verification at the edge is a hosting-layer job, not a Magento one — it belongs in your CDN or WAF. What belongs in the store is the part that is actionable today: knowing which bots you allow, and checking that a request claiming to be one of them really is.
bin/magento angeo:robots:verify-bot-ipangeo/module-robots-txt-aeo manages AI-crawler rules with lossless RFC 9309 parsing and ships the IP-verification command above. IP-range checking remains the practical verification method while signing coverage is partial.
Questions
- What is Web Bot Auth?
- A protocol that lets an automated client sign its HTTP requests with a private key so the origin can verify its identity cryptographically, instead of inferring it from a user-agent string or an IP range. It is being standardised at the IETF and uses HTTP Message Signatures (RFC 9421).
- Is Web Bot Auth a finished standard?
- No. It exists as Internet-Drafts in the IETF Web Bot Auth working group, not as a published RFC, though Cloudflare, AWS and Google have shipped implementations ahead of publication.
- Can I block every bot that does not sign its requests?
- Not safely today. Google states that not all of its agents sign and that origins should keep using IP, reverse DNS and user-agent checks as well. Blocking unsigned traffic would drop legitimate crawlers that have not rolled out signing.
- Does a valid signature mean the bot is trustworthy?
- No. It establishes identity, not intent. A verified operator can still crawl aggressively or ignore your licence terms — the signature just tells you exactly who to hold responsible.
- How is this different from verifying a bot by IP address?
- IP verification depends on the vendor publishing and maintaining accurate ranges, and it breaks when infrastructure moves. A signature is verified from the request itself against a key the operator publishes, so it does not depend on network topology.