llms.txt — Definition, Format and What Actually Reads It
llms.txt is a Markdown file served at the root of a domain that gives AI systems a curated map of a site’s most important content. It is a proposal by Jeremy Howard, published in September 2024, not a standard from any AI vendor. It tells a model which pages matter and what each one is for — the job robots.txt does for permission and sitemap.xml does for coverage, but for meaning.
A plain-Markdown table of contents at /llms.txt, written for language models rather than crawlers, listing the pages that best explain what a site is and does.
What the file contains
The proposal defines a small, fixed structure. Everything below the H1 and blockquote is optional, but the order is not free-form:
# Site name
> One-sentence summary of what this site is.
Optional free paragraph of context.
## Section name
- [Page title](https://example.com/page): One-line description of the page.
## Optional
- [Lower-priority link](https://example.com/x): Skipped when context is short.The ## Optional section is the only heading with defined semantics: a client under context pressure drops it first. Every other section name is yours to choose.
llms.txt, llms-full.txt and llms.jsonl
| File | Contains | Use when |
|---|---|---|
llms.txt | Links plus one-line descriptions | Always. This is the entry point. |
llms-full.txt | The full text of those pages, inlined | The site is small enough to fit a context window. |
llms.jsonl | One self-contained JSON object per line — product, category or page | Ecommerce. A catalogue is data, not prose. |
For a store, the distinction matters more than it looks. llms.txt describes the shop; llms.jsonl describes the inventory. A model answering “does this shop sell waterproof jackets in size L” needs the second, and streaming line-delimited JSON is what lets it read 20,000 products without loading a 40 MB array.
Who reads it — and the honest answer
OpenAI, Anthropic, Google and Perplexity have not published a statement that they consume the file. Anyone claiming a measured ranking effect is overstating what is publicly known. What is true: the file costs almost nothing to serve, is trivially fetchable by any agent given the URL, and several developer-facing platforms publish one.
The defensible reason to publish it is narrower and still real: it is the only artefact where a merchant states, in their own words, what their store is — instead of leaving a model to infer it from theme markup and faceted-navigation URLs.
How common is it on Magento?
We scanned live Magento 2 storefronts drawn from the Tranco top-200,000 list (list ID ZJGPG, frame fixed 8 August 2026). Of the 837 confirmed Magento stores, 762 were reachable in the run of 9 August 2026:
| Signal | Stores | Share of 762 |
|---|---|---|
Serves llms.txt | 85 | 11% |
Emits JSON-LD Product | 73 | 10% |
| Blocks at least one AI crawler | 110 | 14% |
Method and limits: a store counts as serving the file only if /llms.txt returned HTTP 200 with a non-empty body. 23 stores hit rate limits during the run and are counted as not serving it, so 11% is a floor, not a point estimate.
Common mistakes
Publishing it once and never regenerating
A catalogue changes; a static file does not. A stale llms.txt is worse than none, because it states prices and availability that are wrong with the authority of a first-party document.
Letting meta-description artefacts leak in
Most generators build the one-line descriptions from meta descriptions. Whatever is broken there — a truncated first character, an editor note, a character count — is republished verbatim in the file a model quotes.
Listing every URL
It is a curated map, not a sitemap. A 4,000-line llms.txt defeats the purpose, which is to tell a model what to read first.
Generating it on Magento 2
Magento has no native support. angeo/module-llms-txt generates all three files from the catalogue — multi-store, Page Builder aware, atomic writes, MIT-licensed:
composer require angeo/module-llms-txt
bin/magento module:enable Angeo_LlmsTxt
bin/magento setup:upgrade
bin/magento angeo:llms:generate
bin/magento angeo:llms:validateQuestions
- Is llms.txt a standard?
- No. It is a public proposal from September 2024 with no standards body behind it and no ratified specification. Treat it as a convention that costs little to follow, not as a requirement.
- Does llms.txt replace robots.txt or sitemap.xml?
- No — the three answer different questions. robots.txt says who may crawl, sitemap.xml says which URLs exist, llms.txt says which pages explain the site and what each one covers. Serve all three.
- Will llms.txt make ChatGPT recommend my store?
- No, and any vendor promising that is overselling. No AI vendor has confirmed it reads the file. What a merchant controls is crawler access, structured data, a conforming product feed and a verifiable profile — llms.txt is a cheap addition to that set, not a substitute for it.
- Where does the file have to live?
- At the domain root:
https://example.com/llms.txt, served astext/plainortext/markdownwith HTTP 200. A subdirectory copy is not discoverable. - What is the difference between llms.txt and llms.jsonl?
- llms.txt is prose links for humans-as-read-by-models; llms.jsonl is one JSON object per line, one per product, built for catalogue ingestion. A store publishing only llms.txt has described its shop but not its inventory.