llms.txt vs Markdown content negotiation: what agents actually fetch

llms.txt is a Markdown file at the root of a website that tells language models what the site contains and where the clean, text-only versions of its pages live. Markdown content negotiation is a different mechanism for a similar goal: the same URL answers with Markdown instead of HTML when the client sends Accept: text/markdown. Both are often sold as "SEO for AI", and the evidence for what agents actually fetch is thinner than the marketing.

This post compares the two, summarizes the published log studies, and walks through how this site implements both on a static Nuxt build behind plain nginx, without a CDN feature. You get the nginx config, the build step, and a checklist.

What is llms.txt, and what changed in version 2?

llms.txt is a proposal by Jeremy Howard for a Markdown index of a site, served at /llms.txt, with links to Markdown copies of the pages. Version 1 was published on 3 September 2024; version 2 followed on 10 August 2026 (llmstxt.org).

The format is deliberately loose. An H1 with the site's name is the only required part. After it comes a blockquote with a short summary, optional free text, and any number of H2 sections, each a list of Markdown links. A section called "Optional" is, by convention, the list an agent can skip when it needs a shorter context. The proposal also asks for a clean Markdown copy of each page, either at page.md or page.html.md, and index.md for URLs without a file name.

The v2 changes are practical rather than conceptual:

• Discovery through link relations. rel="alternate" type="text/markdown" points from a page to its Markdown copy, and rel="describedby" points to the llms.txt that covers it.
• Both URL styles are allowed: .md appended, or the extension replaced.
• Subpaths can have their own file; "the most specific file applies."
• A simpler consumption model: agents view or search the llms.txt, then follow the links they need. The "Optional" section no longer carries mechanical semantics.

Notably, the proposal does not use HTTP content negotiation at all. It relies on files at known paths and on links.

How does Markdown content negotiation work?

The client lists the media types it wants in the Accept request header, and the server picks a representation of the same resource to return. If the client puts text/markdown first, a server that supports it returns Markdown with Content-Type: text/markdown and adds Vary: Accept so caches keep the two versions apart.

Cloudflare shipped this as a zone feature called Markdown for Agents in February 2026 (changelog). When a request carries Accept: text/markdown, Cloudflare fetches the HTML from the origin, converts it and returns Markdown (Cloudflare docs). The documented details are useful even if you don't use Cloudflare:

• An x-markdown-tokens header estimates the Markdown's size in tokens, and x-original-tokens the HTML's. The documentation's example page drops from 12,345 to 725 tokens.
• Vary gains Accept; ETag and Last-Modified are removed, because conditional requests can't be honored for converted responses.
• If the origin sets no Content-Signal, the default is ai-train=yes, search=yes, ai-input=yes.
• The origin response may not exceed 2 MB, and the feature needs a Pro, Business or Enterprise plan.

Token savings of that size are the real argument for Markdown. An agent that reads your page through a fetch tool pays for navigation, scripts, inline SVG and class names on every call. A Markdown copy is the content and nothing else.

What do AI agents actually fetch?

The published evidence says coding agents increasingly ask for Markdown, and almost nothing reads llms.txt. Both findings come with caveats, but they point the same way.

llms.txt is mostly unread. Ahrefs analyzed server-log data from 137,210 domains with traffic in May 2026 (Ahrefs, June 2026). About 38,000 of them, 28%, had published an llms.txt, yet 97% of those files received zero requests. Of the requests that did arrive, 96% came from bots, led by SEO audit tools; AI retrieval bots made up 1.1%. And zero requests came from AI bots for llms.txt files that don't exist: in Ahrefs' words, "they never go looking."

Some agents do send Accept: text/markdown. In a February 2026 test of seven coding agents, Checkly found three that ask for Markdown first: Claude Code (text/markdown, text/html, */*), Cursor and OpenCode. OpenAI Codex, Gemini CLI, GitHub Copilot and Windsurf sent generic HTML or wildcard headers (Checkly). Agent versions change quickly, so treat this as a snapshot.

One site's logs. A single-site study of Cloudflare's feature counted 1,421 Markdown requests over 44 days (7 March to 19 April 2026), 500 of them from Anthropic's infrastructure and 639 from headless Chrome (Suganthan). The author is explicit that this "does not prove that AI crawlers prefer markdown over HTML" or that serving it improves citations. It's one site; don't extrapolate.

My reading: crawlers that build search and training indexes fetch HTML, as they always have. Agents acting for a user in real time, especially coding agents with a fetch tool, are where Markdown pays off. I don't publish traffic numbers for this site, so none appear here.

How this site implements it with Nuxt and nginx

This site generates a Markdown copy of every page at build time, serves it at a .md URL, and rewrites requests that prefer Markdown to that copy in nginx. The same files feed /llms.txt and /llms-full.txt.

The build step: convert the generated HTML

After nuxt generate has written static HTML, a post-build script (scripts/build-agent-files.mjs) reads each page's <main> element and converts it with the Turndown library. Converting the finished HTML, rather than keeping separate Markdown sources, means the Markdown always says exactly what the page says. A few rules do the real work:

• Scripts, styles, SVG, canvas, buttons and forms are removed. That is why every diagram on this site carries a full text description in its caption: the Markdown copy only keeps words.
• Links and images become absolute URLs, so a copy pasted into a model's context still points somewhere.
• The text follows what a screen reader announces: elements marked aria-hidden are dropped, visually hidden text is kept.
• Each file starts with a short header: the page description, its canonical web URL, its language and the other language versions, and for blog posts the author, dates and keywords.

The same pass writes /llms.txt with sections for pages, blog posts (with dates and keywords), the German and Hungarian versions and an "Optional" list, and /llms-full.txt with every English page in full.

The nginx part: negotiation without a CDN

Two map blocks decide whether a request wants Markdown and which HTML page a Markdown file belongs to:

# Accept: text/markdown listed first, or present without text/html
map $http_accept $bc_md {
  default                             "";
  "~*^\s*text/markdown"               1;
  "~*^(?!.*text/html).*text/markdown" 1;
}

# The HTML page a Markdown copy belongs to (sent as its canonical URL)
map $uri $bc_md_page {
  default                  "";
  "/index.md"              /;
  "~^(?<p>/.+)/index\.md$" $p;
  "~^(?<p>/.+)\.md$"       $p;
}

The HTML location rewrites to the .md file when the map matched, and both locations send Vary: Accept:

location ~ \.md$ {
  default_type text/markdown;
  add_header Link "<https://balazscsorba.com$bc_md_page>; rel=\"canonical\"";
  add_header Vary "Accept";
  try_files $uri =404;
}

location / {
  if ($bc_md) {
    rewrite ^/$ /index.md last;
    rewrite ^/(de|hu)$ /$1/index.md last;
    rewrite ^(/[a-z0-9/-]*[a-z0-9])$ $1.md last;
  }
  add_header Vary "Accept";
  try_files $uri $uri/index.html $uri/ =404;
}

The Link: rel="canonical" header on the Markdown response points search engines back to the HTML page, so the copy doesn't compete with it as duplicate content. One nginx trap is worth knowing: as soon as a location sets its own add_header, it inherits none from the server block. The site keeps its security headers in one include file and pulls it into every location that adds headers.

Discovery and permission signals

• Every page's head links /llms.txt; a small plugin (app/plugins/agent-links.ts) adds a rel="alternate" type="text/markdown" link to the Markdown copy on the top-level pages.
• robots.txt allows all crawlers, lists the AI user agents explicitly and points to llms.txt. The Content Signals line (search=yes, ai-input=yes, ai-train=yes) is kept as a comment, because RFC 9309 validators such as Lighthouse reject unknown directives; the machine-readable permission is /.well-known/tdmrep.json (W3C TDMRep).
• Browser agents with WebMCP can call a get_page_content tool, which fetches the same Markdown copy with Accept: text/markdown. The WebMCP guide covers that layer.

The agent-readiness stack

Think of these mechanisms as layers with different audiences, not as competitors. Each one is cheap on a static site, and each serves a different kind of client.

Trade-offs and pitfalls

The costs are small but real: caching, duplicate content, and a header parser that is a heuristic rather than a full implementation of HTTP negotiation.

• Caches must respect Vary: Accept. Without it, a CDN or proxy can serve Markdown to a browser or HTML to an agent. Check every cache layer between the origin and the client.
• The nginx map ignores q-values. It matches "Markdown listed first" or "Markdown without HTML". That covers the headers Checkly recorded, but a client sending text/html;q=0.1, text/markdown gets HTML. A full parser belongs in application code if you need one.
• Duplicate URLs. The .md copy is a second URL for the same content. A canonical Link header handles search engines; don't list the copies in your sitemap.
• Drift between versions. Hand-maintained Markdown goes stale. Generating it from the built HTML removes the problem at the cost of a build step.
• Don't expect llms.txt to move rankings. The Ahrefs data says it is mostly unread. Publish it because it is cheap and useful to the tools that do read it, not as a ranking bet.

Checklist: llms.txt and Markdown for agents

1. Generate Markdown from the rendered HTML, one file per page, with absolute links.
2. Describe every diagram in words; converters drop SVG and canvas.
3. Serve the copies as text/markdown with a canonical Link header to the HTML.
4. Negotiate on Accept at the same URL and send Vary: Accept on both representations.
5. Publish /llms.txt with a summary blockquote and one link per page, and an "Optional" list.
6. Add rel="alternate" type="text/markdown" links in the HTML head, as llms.txt v2 suggests.
7. State permissions in robots.txt and a machine-readable file such as TDMRep.
8. Measure your own logs before claiming anything about agent traffic.

The next layer up is letting agents act, not just read: see the guide to WebMCP on a real site, and for shops, the comparison of agentic commerce protocols. If you want this set up on your own site, see AI engineering.

Sources

1. llmstxt.org: The /llms.txt file (proposal, version 2)
2. llmstxt.org: Changes from v1 to v2
3. Cloudflare changelog: Markdown for Agents (12 February 2026)
4. Cloudflare docs: Markdown for Agents
5. Ahrefs: 137K sites analyzed, 97% of llms.txt files never get read (June 2026)
6. Checkly: The current state of content negotiation for AI agents (February 2026)
7. Suganthan: tracking Cloudflare Markdown for Agents on one site