Verified AI agents with Web Bot Auth

“Block all bots” is now a business decision#

More than half of web traffic is automated, and a growing share of it is legitimate: AI agents shopping, booking and researching on behalf of real people. An agent filling your checkout with a customer’s card is revenue; a scraper farming your catalogue is not. To a naive bot filter, they look identical: both score low on any honest humanity test, because both are automation.

The distinction that matters is no longer human versus bot. It’s automation that declares who it is versus automation that hides. Edge Shield is built around exactly that line.

How Shield identifies agents#

The primary mechanism is Web Bot Auth, the emerging IETF standard (RFC 9421 HTTP Message Signatures with the web-bot-auth tag) backed by OpenAI, Google, Amazon and Cloudflare. The agent signs each request with an Ed25519 key and points to its published key directory. Shield fetches the directory, checks the signature and confirms exactly which agent is calling. That identity can’t be spoofed by copying a user-agent string.

GET /products HTTP/1.1
Host: example.com
Signature-Agent: "https://agent.bot.goog"
Signature-Input: sig1=("@authority" "signature-agent");created=1753100000;\
  keyid="poqkLGiy...";alg="ed25519";expires=1753100300;nonce="e8N7...";tag="web-bot-auth"
Signature: sig1=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQdi5N5YYKrD+mCT1HA1nZDsi6nJKuHxUi/5Syp3rLWBA==:

Established crawlers that predate the standard (Googlebot, Bingbot, Applebot) are confirmed the way their operators document: reverse-DNS lookup of the calling IP, forward-confirmed back to the same address. A client merely claiming to be Googlebot from an unrelated IP gets no credit, and the lie counts against its humanity score.

Verified identity is bound into the Shield response token and surfaced in the siteverify response as the agent field.

Set a per-widget agent policy#

Each Shield widget chooses how verified agents are treated, in its settings:

Whatever the policy, anonymous automation is always treated aggressively: clear automation signals with no declared identity hard-cap the humanity score. Declaring identity is the fast path; that’s the incentive for agents to sign their traffic.

Route on identity as well as score#

The score answers “is this a human?”; the agent field answers “then who is it?”. Use both in your server-side check:

const result = await siteverify(token)

if (!result.success) reject()
else if (result.agent?.verified) {
  // Declared automation: route it, don't fight it
  routeToApi(result.agent.domain)
} else if (result.score >= 40) {
  allow() // human traffic
} else {
  block() // automation that refused to identify itself
}

Note that agents don’t only arrive through forms. When a signed request hits your API or a content page directly, forward its signature headers to Shield’s agentverify endpoint and get the same verified identity back, without a widget, JavaScript or a challenge. See Protecting APIs from bot abuse for the pattern.

One caution: Web Bot Auth adoption is early and uneven. Google signs its AI-browsing agent but not Googlebot indexing, and rollouts are gradual. Treat a valid signature as strong positive identity; treat its absence as neutral, not negative.

The other direction: your agents on Edge#

If you also run AI agents (deploying infrastructure, managing DNS), the same principle applies in reverse: give automation scoped, revocable identity rather than full credentials. On Edge that means agent access codes: per-agent credentials with product restrictions, budget caps and expiry dates, so a leaked agent context never exposes your whole account.