Agent Readiness CompareA square mark containing five stacked bars, the fourth in ultramarine.Agent Readiness Compare

DISPATCH · 2 SEP 2026

Six agent readiness myths, checked against the specifications

Each claim, what the source says, and what to do instead.

Agent Readiness Compare editors · Spec status checked September 2026

Agent Readiness Compare editors · · 5 min read

Answer

llms.txt does not control crawling, robots.txt does not enforce anything, a user agent string proves nothing, WebMCP is not a version of MCP, most sites do not need x402, and a readiness score does not measure AI visibility or revenue. Each section below cites the page that settles the point.

On this page
  1. 1. Myth 1: llms.txt controls what AI crawlers can fetch
  2. 2. Myth 2: robots.txt blocks AI agents
  3. 3. Myth 3: a user agent string tells you which bot is visiting
  4. 4. Myth 4: WebMCP is the browser version of MCP
  5. 5. Myth 5: every agent-ready site needs x402
  6. 6. Myth 6: a high readiness score means more AI visibility or more sales
  7. 7. Why do these myths persist?
  8. 8. Sources

1.Myth 1: llms.txt controls what AI crawlers can fetch

It does not. llms.txt, proposed by Jeremy Howard in 2024, is a markdown file with a required H1, an optional summary and lists of links; its job is to point language models to the pages that matter. Crawl permissions live in robots.txt. v2 of the proposal, dated 10 August 2026, adds link relations but still does not grant or block access. What to do: publish both files and keep them consistent. See what is llms.txt?

2.Myth 2: robots.txt blocks AI agents

robots.txt asks; it does not enforce. RFC 9309 states that its rules "are not a form of access authorization." Well-behaved crawlers follow them, and others may not. What to do: use robots.txt to state your policy, and enforce what matters at your server or CDN, for example with Cloudflare AI Crawl Control or verified bot rules. See robots.txt for AI crawlers.

3.Myth 3: a user agent string tells you which bot is visiting

Any client can send any User-Agent header, so a rule that allows GPTBot by name also allows anything that claims the name. Verification needs a signature. Web Bot Auth builds on HTTP Message Signatures (RFC 9421), with the operator's public keys in a key directory at /.well-known/http-message-signatures-directory. Cloudflare documents it as a bot verification method, and Vercel's bot verification supports it. See how websites recognise AI agents.

4.Myth 4: WebMCP is the browser version of MCP

They share the idea of described tools, but they are separate efforts. MCP is a protocol between hosts, clients and servers using JSON-RPC 2.0, with a versioned specification whose latest revision is dated 2026-07-28. WebMCP is a W3C Web Machine Learning Community Group proposal that lets a page register tools, currently through document.modelContext, which an in-browser agent calls using the user's session. What to do: choose by where the task lives. See MCP vs WebMCP.

5.Myth 5: every agent-ready site needs x402

x402 fits resources an agent should buy per request, such as per-call APIs or datasets: the server answers an unpaid request with HTTP 402, and the client pays and retries. Most subscription software is bought by companies on contracts, invoices or cards, so readable pricing and a checkout or demo request an agent can complete matter more. What to do: decide per resource and record the decision (checklist L4.4). See what is x402?

6.Myth 6: a high readiness score means more AI visibility or more sales

A readiness score summarises which signals a scanner found. It is not designed to measure how often you appear in AI answers, and we have seen no public evidence that it does. No scanner measures sign-ups or revenue either. What to do: treat readiness, AI visibility and business outcomes as three separate measurements, and attribute outcomes with your own analytics. See agent readiness vs AEO and how to read a readiness score.

7.Why do these myths persist?

Mostly because the files and protocols arrived quickly and look alike: two text files at the site root, two ways to expose tools, and several scores with similar names. The fix is to go back to the source each time. Every standard page on this site states its maintainer, its status and a link to the specification, and the FAQ collects the short answers.

8.Sources

Reviewed Sep 2026