In July I scanned a set of live Magento storefronts for four signals that affect how legible a store is to an AI assistant. In August I scanned a larger set built the same way. 376 stores appear in both runs and answered both times. Two things came out of pairing them: the signals barely moved at all, and where crawler rules did move, no store changed its mind about one crawler — it changed several at once, every time.
Why measure the same stores instead of a new sample
Two aggregate percentages moving from 13% to 15% tells you very little. The same shift is produced by six stores improving, by sixty improving while fifty-four regress, or by a slightly different sample. The only way to tell those apart is to hold the stores fixed and look at each one.
462 domains were scanned in July and 770 in August, both drawn from the Tranco list. 382 domains are in both files, and 376 returned a usable result in both runs. That set of 376 is the panel, and everything below is about them.
A store that answered in one run and not the other is excluded. Not observing a signal is not the same as observing its absence, and treating a timeout as a missing llms.txt would quietly inflate every number here.
Finding 1 — 94% of the panel did not change at all
354 of the 376 stores showed no change in any of the signals or crawler rules measured. Twenty-two stores changed something.
That is a claim about four checks and eight crawler rules, not about the sites. A store can have replaced its theme, its catalogue and its hosting in between and still sit in that 354.
| Signal | July | August | Gained | Lost |
|---|---|---|---|---|
Serves llms.txt |
50 (13%) | 55 (15%) | 6 | 1 |
| Blocks an AI crawler | 40 (11%) | 41 (11%) | 3 | 2 |
JSON-LD Product * |
19 (11%) | 18 (10%) | 0 | 1 |
offers.availability * |
16 (9%) | 15 (9%) | 0 | 1 |
* over the 173-store sub-panel explained in the next section, not the full 376.
llms.txt is the only signal with a net directional movement in this panel: six stores added it, one dropped it. Six out of 376 is not a trend, and I have not tested whether it is distinguishable from chance. It is simply the only column where the two directions are not roughly balanced.
Finding 2 — the sampling catch, which changed one of the answers
The two schema signals depend on which product page the scanner happened to sample. It finds one by following a candidate product link from the homepage — a link whose URL or text matches common product-path patterns — and that heuristic does not always land on the same page twice.
Of the 196 stores where a product page was found in both runs, 23 — 12% — had a different page sampled the second time. At one retailer the scanner tested a bicycle in July and a paperback in August. At another it tested a bicycle, then a delivery-information page.
Compared naively, both of those stores appear as lost their JSON-LD markup. Neither changed anything. The apparent loss is the scanner looking somewhere else.
So the schema figures above are computed only over the 173 stores where the identical URL was tested in both runs. On that stricter footing the JSON-LD result is cleaner and duller: zero stores gained Product markup and one lost it, and that one is an online course provider rather than a shop. The offers.availability figure moves with it, for the same store and no other. Microdata was unchanged in both directions — 26 stores before, 26 after.
If you do this kind of before-and-after work, pin the URL. Otherwise the sampling heuristic can manufacture changes that never happened, and at 12% here, often enough to swamp the real signal.
Finding 3 — nobody changed their mind about one crawler
Each store’s robots.txt was checked against eight AI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and Google-Extended. A rule counts whether it names the agent or the agent inherits it from a wildcard group.
Nine stores in the panel changed at least one of those rules between the runs. This is how many agents each of them moved:
Not one of the nine changed a single agent on its own. All nine moved GPTBot and ClaudeBot together. Four rewrote all eight rules at once. The smallest edit anyone made touched two agents.
In the July write-up I noted that GPTBot and ClaudeBot are blocked at four times the rate of PerplexityBot and suggested this looked more inherited than decided. That was an inference from a cross-section, and a reader was entitled to answer that operators might simply hold different views of different crawlers.
The panel does not settle that, but it constrains it. Over those sixteen days it shows no store making an independent, crawler-by-crawler change. Every observed change touched several agents at once, which is more consistent with bundled policy edits — a pasted robots.txt snippet, a plugin updating its bot list, a security vendor changing a default — than with per-crawler judgements.
It does not rule per-crawler policy out. Nine stores over sixteen days is a thin base, and a store can hold a considered position without editing it in the window I happened to watch. What it does mean is that anyone arguing these are deliberate per-crawler decisions now has to account for nobody making one.
What this is good for
A baseline that does not move is worth more than a baseline that does.
The next study in this series asks whether businesses carrying these signals are named more often by AI assistants than comparable businesses without them. That question has an obvious objection: stores change, assistants change, and any difference you find might be drift.
This measurement puts a number on that. Across the panel, 94% of stores showed no change in any measured signal, and the schema signals moved in no store at all where the comparison was clean. Whatever a visibility measurement turns up, large-scale reconfiguration of these particular signals is unlikely to be the explanation.
What this does not show
The two samples overlap, so this is not an independent replication. 382 of the 462 July domains are in the August frame. The stronger evidence of measurement stability is not that the two runs’ aggregate percentages agree — it is that 354 individual stores returned identical results in both.
The window is short. Sixteen days. A market that looks frozen over two weeks may well move over a year, and “nothing changed” is a much weaker claim at this timescale than it would be at that one.
This is not a representative sample of Magento stores. Both frames are drawn from the top of Tranco, so the sample is biased toward more prominent sites. Small independent stores may behave differently — and they are also the ones with the most to gain from being findable.
Each run is a single observation of a live site. A store that was rate-limited, served from a CDN edge, or rendering its structured data client-side can look emptier than it is. Twelve stores in the panel encountered rate limiting during the August run; they still returned a usable result and are included rather than dropped, so their values are less reliable than the rest.
Method and reproduction
The July scan ran on 24 July 2026 and the August scan on 9 August 2026, the latter over a frame drawn from Tranco list ZJGPG, top 200,000 domains. Both runs, the domain-level comparison, the list IDs, the exclusion file and the classification change log are published.
The scanner identifies itself in every request, obeys robots.txt including Crawl-delay, waits at least two seconds between requests to the same host, keeps no page content, and can be blocked in two lines. The crawler policy has the details, including why no individual store is named in this article.
Previous instalment: the July baseline across 462 stores.