UNIVERSITY · LESSON 3 · BASICS · 3 MIN READ
How websites recognise AI agents: user agents, robots.txt and signed requests
Three signals, and which of them is actually verification.
Agent Readiness Compare editors · Spec status checked September 2026
Answer
A site has three ways to tell which bot or agent is visiting: the user agent it claims, the rules it is asked to follow in robots.txt, and cryptographic signatures that prove who sent a request. Only signatures are verification; the other two rely on the client being honest.
On this page
1.What does the user agent tell you?
Every HTTP request can carry a User-Agent header, such as GPTBot or ClaudeBot for AI crawlers. It is a claim, not proof. Any client can send any string, so user-agent rules alone cannot separate a real AI crawler from automation that copies its name.
2.What does robots.txt do, and not do?
robots.txt, standardised as RFC 9309 in September 2022, lets a site publish path rules per user agent, and well-behaved crawlers follow them. The RFC states that "these rules are not a form of access authorization": robots.txt asks, it does not enforce. On 15 September 2026 Cloudflare added a setting that uses robots.txt Disallow rules to opt out of AI training while staying in search, relying on crawler operators it designates as "Accountable" to honour it.
3.How do signed requests work?
Web Bot Auth builds on HTTP Message Signatures (RFC 9421). The bot operator publishes public keys in a key directory at a well-known path and signs each request; the site, or its CDN, checks the signature against those keys. Cloudflare's documentation lists the Internet-Drafts it implements and uses Ed25519 keys. Vercel's bot verification also supports Web Bot Auth, so signed bots can pass its bot protection. Since 28 August 2026, Cloudflare's BotBase for Operators validates Web Bot Auth signatures automatically when an operator submits a bot.
4.What should a site do with a verified agent?
Verification only answers who sent the request; the site still decides what that agent may do. Common choices are to let verified agents through a bot challenge, rate-limit them separately from unknown automation, and log them for analytics. Vercel's BotID shows the pattern in code: checkBotId() returns isVerifiedBot and verifiedBotName, so a handler can allow a named verified bot, such as 'chatgpt-operator', and block others. Lesson 9 covers the policy side.
5.What about agents acting for a signed-in user?
A signature identifies the operator, not the person. When an agent acts inside a user's account, access is usually delegated with OAuth 2.0 scopes. The MCP specification of 28 July 2026 tightened its authorization rules, and the MCP roadmap of 22 August 2026 lists agent identity as a priority.
See Web Bot Auth and robots.txt for AI crawlers. Next track: testing tools and fixing tools.
Source: RFC 9309 · RFC 9421 · Cloudflare Web Bot Auth docs · Vercel verified bots · Cloudflare, BotBase for Operators · Cloudflare, stay discoverable in search while disallowing AI training · MCP blog · Reviewed Sep 2026