Agent Readiness CompareA square mark containing five stacked bars, the fourth in ultramarine.Agent Readiness Compare

UNIVERSITY · LESSON 2 · BASICS · 3 MIN READ

How AI agents read a web page: HTML, markdown and llms.txt

What arrives in the response, and three ways to make it easier to read.

Agent Readiness Compare editors · Spec status checked September 2026

Answer

An agent reads what your server returns. If key facts appear only after JavaScript runs, many agents never see them. Server-rendered HTML is the baseline; a markdown version of each page and an llms.txt file make reading cheaper and more predictable.

On this page
  1. 1. What does an agent actually receive?
  2. 2. What is markdown negotiation?
  3. 3. What does llms.txt add?
  4. 4. What about structured data?
  5. 5. What should you check first?

1.What does an agent actually receive?

When an agent fetches a URL it gets the HTTP response: headers and a body. Some agents render JavaScript in a browser; many fetchers do not. A pricing table injected by client-side code may be empty in the response the agent reads. Server-rendering or prerendering pricing, plans, docs, contact and sign-up pages removes that risk.

2.What is markdown negotiation?

A client can ask for markdown by sending Accept: text/markdown. A server or CDN that supports it returns a markdown version of the page instead of HTML. Cloudflare's Markdown for Agents does this conversion at the edge; its launch post of 12 February 2026 describes the request header and an x-markdown-tokens response header with an estimated token count. Markdown drops layout markup, so the agent spends its context on content.

3.What does llms.txt add?

llms.txt is a markdown file at the site root with a title, a short summary and lists of links to the pages that matter most. It is a curated index, not a sitemap. The proposal was revised to v2 in August 2026: pages can now point to their markdown version (rel="alternate" type="text/markdown") and to the llms.txt that covers them (rel="describedby"), both page.html.md and page.md URL forms are accepted, and an llms.txt in a subpath covers the pages under that path.

4.What about structured data?

JSON-LD states facts in a form machines parse without guessing: the organization, its products, and offers with prices. It sits in the page's HTML, so it survives when an agent cannot run JavaScript, and Cloudflare's Markdown for Agents keeps JSON-LD as a fenced code block when it converts a page. It does not replace readable text: prices and plan limits should also appear in the page body, not only in images, sliders or toggles (checklist L2.4 and L2.5).

5.What should you check first?

  1. Fetch your pricing page with JavaScript disabled and confirm the plans and prices are in the HTML.
  2. Request a key page with Accept: text/markdown and look at what comes back.
  3. Open /llms.txt and confirm every link resolves to a current page.

The llms.txt standard page and the read-layer checks have the detail. Next: how websites recognise AI agents.

Source: llmstxt.org · llms.txt changes · Cloudflare, Markdown for Agents launch post · Cloudflare Markdown for Agents docs · Reviewed Sep 2026

Back to University