lms.txt v2: What the Spec Says, and What the Evidence Shows

Checked against llmstxt.org v2, revised 10 August 2026, and against the published request data as of September 2026.

We shipped a validator that could certify a broken llms.txt file as valid.

The bug was simple. The spec recommends a short summary in a blockquote under the title. Our checker looked for a line starting with >. It never looked at what came after it. A store with no meta description produced a bare > and nothing else, and our tool ticked the box.

Fixing it sent us back to two things we had not read carefully enough. The specification had been revised on 10 August 2026, quietly, without a corresponding launch announcement. And Ahrefs had published server-log data from 137,210 domains showing that almost nobody requests these files at all.

Both matter, and most writing about llms.txt engages with neither. This article covers the evidence first, then the spec, then four claims about the spec that do not survive a check against the source. One of those four was ours.

What the request data shows

• 97%of valid llms.txt files received zero requests in May 2026
• 137,210domains in the Ahrefs sample
• 19.5%of the fetches that did happen came from named AI tools
• ~55%of live files have the recommended shape (separate sample)

In June 2026 Ahrefs published a study of every domain in its Web Analytics data that received traffic in May 2026 — 137,210 of them. It checked each domain root for an llms.txt returning HTTP 200, confirmed the file was real Markdown rather than an HTML error page, and then looked at every request to those paths in its bot analytics.

The headline numbers:

• 28% of those domains publish an llms.txt. Ahrefs attaches its own caveat: its customers skew more technical and SEO-aware than the web at large, so 28% is an upper bound, not a web-wide figure.
• Of roughly 38,000 valid files, 97% received zero requests in May 2026. Not few requests. None.
• Only about 1,100 files received any traffic at all.
• AI bots never requested llms.txt on domains where it did not exist. They are not probing for it. A crawler that wanted the file would ask and collect 404s.

The breakdown of the 3% that were fetched is more interesting than the headline, and it cuts both ways.

About 19.5% of fetches came from named AI tools, with GPTBot first and Claude-Code second — ahead of every AI search and assistant bot. Roughly 12% came from the industry inspecting itself: GEO and AEO tools, llms.txt checkers, researchers. Chrome’s Lighthouse llms.txt audit accounted for about one fetch in a thousand.

Read that carefully, because it is easy to quote in either direction. The file is not universally ignored — where it is fetched, a fifth of the fetches are named AI tools. But the tools at the top are coding assistants reading documentation, not the retrieval crawlers that decide whether a shop gets cited in a shopping answer. Separate reporting of the same dataset puts AI retrieval bots at around 1% of total requests.

That pattern matches what Google has said. John Mueller described llms.txt in June 2025 as not being done for search, closer to a stopgap that saves tokens for AI coding tools reading developer documentation. In June 2026, Google added a note to its AI optimisation guidance stating that these machine-readable text files neither help nor harm Search rankings, because Search ignores them.

There is a second finding worth more than the adoption headline. In June 2026 Chris Humphrey fetched /llms.txt for the top 10,000 Majestic Million domains. Of the 1,050 that returned HTTP 200, 313 were serving an HTML soft 404 rather than a file. And of the 737 real files, only about 55% had the recommended shape: a title, a summary, and at least one section of curated links.

Nearly half of the published files are malformed. Hold that thought — it comes back later.

What changed in llms.txt v2

Against that background, the specification was revised. From its own change notes:

• Link relations were added. A page can declare where its Markdown version lives, and which llms.txt describes it. This is the main addition.
• Both Markdown URL forms are allowed. The earlier version described appending .md to the full page URL (page.html.md); v2 also allows replacing the extension (page.md).
• Files at subpaths are defined. An llms.txt covers the pages under its own path, and the most specific file wins.
• ## Optional lost its mechanical semantics.
• Links should lead to LLM-friendly content, meaning the Markdown versions of pages where those exist.

Now the four claims about all this that do not hold up.

Mistake 1: “llms-full.txt is part of the spec”

Short answerllms-full.txt is not defined by the llms.txt specification.

The spec does not define it. The companion files it describes are llms-ctx.txt and llms-ctx-full.txt, generated by the llms_txt2ctx tool — and that context-expansion mechanism is no longer defined by the v2 specification either.

llms-full.txt — one large file holding the full text of a site — grew out of that pattern and was popularised when Mintlify rolled it out platform-wide in November 2024. Different name, different mechanism. People are half-remembering something that existed, which is why the mistake is sticky.

We got this wrong ourselves. Our audit module told merchants with large files to “split into llms.txt + llms-full.txt per spec.” That sentence made a claim about a specification that the specification does not make. It is fixed.

The distinction was already documented publicly. Stack Exchange’s design system made exactly this point in a pull request in May 2026 while splitting its own files, and others have since. We are not adding a new fact here, only repeating one that has not displaced the wrong version.

“Not part of the spec” does not mean “useless.” The correct claim is not don’t publish it. It is don’t call it part of the specification, and know that support for it is an ecosystem convention.

Mistake 2: “There is a W3C draft”

Short answerW3C issue #506 is not a W3C Working Draft.

Several articles published in mid-2026 describe a June 2026 W3C proposal to standardise llms.txt. They tell readers to audit their file against “the W3C draft’s strict Markdown subset,” and to submit comments to “the W3C working group.” Some add that the draft forbids llms.txt at subpaths and requires a version header.

The W3C item people appear to be referring to is issue #506 in the w3c/strategy repository, opened by xfq on 27 April 2025, labelled AI, and still open. Its own text says the proposal is at an early stage and does not introduce any new format. That is a request to consider a topic, not a working draft.

The subpath claim also contradicts the actual specification. v2 explicitly permits files at subpaths — /llms.txt, /docs/llms.txt, /de/llms.txt — and explains how they resolve: a file covers the URLs under its path, and where several apply, the most specific wins. That is what makes a per-language file valid on a multi-store setup.

You can check this in under a minute. Open the issue, read the first paragraph, compare it to the article’s claim.

Mistake 3: “## Optional tells the AI what it can skip”

Short answer## Optional is now a convention, not a mechanical omission rule.

This one needs care, because the correction is easy to overshoot.

In the earlier version, ## Optional had a defined mechanical meaning: it was used by the context-expansion mechanism to determine which URLs could be omitted. v2 removed that mechanism, and the special meaning went with it.

What remains is real but weaker. The spec still describes ## Optional as a convention for secondary information — links an agent may skip when a shorter context is needed. So it is not meaningless. It is no longer a rule any parser is expected to implement.

The practical difference for a shop: many Magento llms.txt files put the whole product list under ## Optional, on the theory that a context-constrained model will drop the catalogue and keep the categories. That theory rested on the mechanical semantics, and those are gone. Products are the primary content of a store, and secondary is precisely what they are not.

We changed our own default for this reason.

Mistake 4: “There’s a blockquote, so the summary is fine”

Short answerAn empty > is not a meaningful summary.

This is the one worth reading twice, because it is not about the spec. It is about what validation misses — and, given that nearly half of published files are malformed, it is not a rare problem.

The spec recommends a short summary in a blockquote directly under the H1, containing key information about the site. The H1 is the only element it requires; the summary is recommended, not mandatory. That distinction matters, and not in the direction you might expect: an omitted summary is a defensible choice, while an empty blockquote claims to be one and is not.

Our validator checked for a line beginning with >.

A Magento store with no meta description and no custom summary produced this:

# Demo Store

>

## Categories

An empty blockquote. Our validator reported the file as valid. So did our audit module. Both saw a blockquote marker, counted it, and moved on. The file was broken in the exact place the check existed to protect, and the check was the thing confirming it was fine.

The failure is easy to name once you see it: the checker tested for the shape of the summary rather than its content. A parser that accepts the Markdown structure can therefore encounter an empty blockquote where the summary should be — worse than no summary at all, because an absent summary is at least visibly absent.

Both our tools now treat a bare > as an error. If you run any llms.txt validation, this is worth five minutes: put an empty blockquote in a test file and see whether your tool complains.

What v2 asks you to add

The link relations are the substantive new work. The point is that an agent holding a page URL should not have to guess where the Markdown version lives.

In the page head:

<link rel="alternate" type="text/markdown" href="https://shop.example/blue-shirt.html.md">
<link rel="describedby" href="https://shop.example/llms.txt">

For the Markdown resource itself, the spec describes the HTTP Link: header, since that resource has no HTML head to carry tags:

Link: <https://shop.example/llms.txt>; rel="describedby"

Three things to get right beyond that:

1. Support both URL forms if you can. v2 permits page.html.md and page.md. Serving either one is compliant. Supporting both is a compatibility choice for implementations that may assume either form.
2. Point llms.txt links at the Markdown versions where you serve them.
3. Do not declare a relation that 404s. A broken rel="alternate" can be worse than no relation at all, because a client may follow it and fail.

How we found out we were wrong about our own store

Our audit module runs against our demo store on every release. After we shipped v2 support, the new link-relations check failed — on our own store.

The cause was configuration, not code. Markdown mirrors were switched off, so the head tags were correctly suppressed and the .md URLs correctly returned 404. One setting. Once enabled, the check passed.

We include this because the alternative — writing the post after quietly fixing it — leaves out the useful part. A checker that has only ever shown green on the author’s own site has not been tested against anything.

What to check on your store

Five commands. Replace the host and the product URL with your own.

# 1. Fetch the file and inspect its opening lines
#    -f makes curl fail on an HTTP error instead of piping the error page
curl -fsS https://shop.example/llms.txt | head -3

# 2. No empty blockquote — this should print nothing
curl -fsS https://shop.example/llms.txt | grep -n '^>[[:space:]]*$'

# 3. Link relations in the page head (presence check, not full validation)
curl -fsS https://shop.example/blue-shirt.html \
  | grep -i 'rel="alternate"\|rel="describedby"'

# 4. Link header on the Markdown file — GET, not HEAD, because caching
#    layers do not always return identical headers for a HEAD request
curl -fsS -D - -o /dev/null https://shop.example/blue-shirt.html.md \
  | grep -i '^link:'

# 5. Markdown URL forms
curl -so /dev/null -w '%{http_code}\n' https://shop.example/blue-shirt.html.md
curl -so /dev/null -w '%{http_code}\n' https://shop.example/blue-shirt.md

What to expect: command 2 prints nothing; command 3 shows both relations; command 4 shows a Link: header; command 5 returns a successful response for each Markdown URL form that you serve — in our implementation, 200. If you support only one form, the other may legitimately return 404. Command 5 tests what your implementation supports, not whether it is compliant.

Command 3 is deliberately loose. It finds the relations but does not verify that rel="alternate" also carries type="text/markdown", so a malformed tag will still match. Read the output rather than trusting the exit code.

If you want command 2 in CI, remember that grep exits 1 when it finds nothing, which is the passing case here. Invert it:

if curl -fsS https://shop.example/llms.txt | grep -q '^>[[:space:]]*$'; then
  echo "ERROR: empty summary blockquote"
  exit 1
fi

So should a Magento store implement this?

Not as a traffic tactic. The request data does not support it, Google states plainly that Search ignores the file, and no major model provider has committed to reading it in production. If someone sells you llms.txt as an AI-visibility lever, ask them for request logs.

Possibly as developer-facing infrastructure. The one place the data shows real consumption is coding assistants reading documentation — GPTBot and Claude-Code lead the fetches. If you publish API documentation or a developer portal alongside your store, that use case is measurable today.

Reasonably, as cheap optionality. A generated file costs a build step and stays current automatically. The failure mode to avoid is not publishing one; it is publishing a stale, hand-maintained one, or a broken one — and given that roughly half of live files are malformed, that risk is the realistic one.

And do not let it displace the work that does have evidence behind it. For a shop, structured product data, correct canonical and schema markup, and pages an agent can actually parse have measurable effects. llms.txt does not, yet.

The most useful thing v2 changes for a merchant is not visibility. It is that the file, the Markdown mirrors and the link relations together give an agent a defined way to move from a page to a machine-readable version of it. That is worth having ready. It is not worth overselling.

What we do not know

• We have no evidence that AI systems or agents follow the v2 link relations. They are new, correct per the spec, and cheap to serve. Whether anything follows them is unmeasured, including by us.
• Adoption figures depend entirely on the population sampled. Ahrefs found 28% among a technical, SEO-aware customer base and said so itself. Humphrey found 7.4% in the Majestic top 10,000. As he put it: there is no single universal adoption rate, only the rate inside a defined slice.
• The request data is one month, one vendor’s panel. May 2026, Ahrefs Web Analytics. It is the largest public dataset of its kind and it is still one month of one panel.
• Absence of observed consumption is not proof of no value. It is what the logs currently show. That could change if a major provider commits to reading the file — and the v2 link relations exist partly to make that easier.
• v2 may not be the last revision. The first version stood for nearly two years before the specification changed. If you build tooling against it, build in a way to notice when it moves.

We maintain an open-source Magento 2 module that generates these files, and an audit module that checks them. Both are MIT licensed. The two bugs described above were in our own code and are fixed in the current releases.