AI assistants named 458 shops

Two earlier pieces measured how Magento stores are configured for AI assistants: a baseline scan, since repeated on a larger frame, and a store-by-store repeat that found 94% unchanged. Both measured configuration. Neither measured visibility. This one asks whether four observable technical signals distinguish businesses that assistants name repeatedly from businesses they name once. None of the four was more common among the repeatedly named group by the margin I pre-registered. One of them, llms.txt, was significantly less common there — the opposite of what the field assumes. I wrote down what I expected before I had the data, hashed it, and sent it to the other party to hold — what follows is that prediction against the result, including the parts where I was wrong.

The question

Do businesses that AI assistants name repeatedly differ, on measurable technical signals, from businesses the same assistants name exactly once?

Note what that does not ask. Every business in this study was named at least once, so nothing here estimates the chance of being named at all. It estimates the difference between repeated and one-off mentions among businesses already in the answers.

Four signals, all visible from outside a store: whether robots.txt blocks any of eight AI crawlers; whether the site serves an llms.txt; whether a product page publishes JSON-LD Product; and whether that node carries offers.availability.

Where the data comes from, and what I did not control

The answers are not mine. They come from connexion.me, who ran a paired home-decor study across ChatGPT, Gemini and Perplexity for this one: 44 product-level buying questions asked twice, once plain and once with a constant extra sentence, giving two arms of 264 answers each (44 questions × 3 engines × 2 runs). Questions like where can I buy blackout curtains for a bay window, three drops to one length — specific enough that an engine has to reach past the marketplaces.

One thing to fix in the reader’s head before any number: this measures which businesses these three engines named, under this question set, on these days. That is not the same as AI visibility in general, and nothing below should be read as though it were.

I did not write the questions. I never saw the store list they produced until after my analysis plan was sealed, and they never saw my sample frame, my scan results or my thresholds. That blinding is deliberate: if either of us could see the other’s data, someone could reasonably say the questions were written toward the shops or the thresholds toward the answers.

Their roster carries 669 businesses, the union of both runs and both arms, with per-run mention counts and, where resolvable, a domain.

The paired home-decor study is not on their public site yet — by agreement it goes up after this piece. Their published boards, which document the same method on a different category, are at connexion.me/c/crmctx; the figures on that page are theirs and are not used here.

What that leaves me unable to check. The engines’ answers, the extraction prompt, the name-folding rules and the domain resolution are all theirs. I re-cut their normalisation from the pre-normalisation strings they ship and got their published counts back, which is a check on the arithmetic, not on the instrument.

What I predicted, before any of it

Sealed 10 August, SHA-256 9b4ccf12629e…, amended twice, both amendments sent and hashed before the data they concern existed. The predictions:

Under 15% of named businesses would be running Magento. My frame was Magento stores from the top of Tranco; the businesses an engine names for a bay-window blind are not that.

No signal would separate the groups by more than 15 percentage points. My reasoning was my own baseline: JSON-LD Product sits at 10% and llms.txt at 11% across 762 Magento stores, and in a 376-store panel measured twice, not one store added product markup. A market where almost nobody does the thing cannot show much difference between those who do and those who do not.

I would be wrong if any of the four differed by 20 points or more with the named group higher.

Those are two different thresholds and it matters which is which. The 15 points was my expectation. The 20 points with the named group higher was the pre-registered failure criterion, written that way because the study exists to test whether these signals help, and a difference in the other direction would not answer that question.

And, in the document: a null result is a result and gets published as one.

How the groups were built

Cases and controls both come out of the same corpus: every business in either group was named in at least one answer. The grouping variable is mention frequency — cases were named repeatedly, controls exactly once.

• Head excluded first: anything named in 53 or more of the 264 answers in that arm — 20%. In the generic arm that removes Amazon, Etsy, Wayfair, Target and Home Depot; in the context arm, Amazon and Etsy.
• Cases: a total of three or more mentions across the two runs combined, and present in both of them. The presence rule matters: of the businesses named exactly once in the first run, only 42% were named at all in the repeat.
• Controls: a combined total of exactly one mention across the two runs.
• Both counts are aggregates over the pair of runs, not per-run figures.

Then the losses, which are large and which I would rather state than bury:

Group sizes: 58 cases against 196 controls in the generic arm, 53 against 139 in the context arm.

That leaves most of the 455 unaccounted for, so here is where they all go. Each column independently accounts for all 455 scanned domains. Do not add the Generic and Context columns together. A business can be a case in one arm and neither in the other.

The case and control counts are arm-specific samples from the same 455-domain universe. They are not mutually exclusive subsets and must not be added across arms. The universe is the union of businesses appearing in either arm, so “never named in this arm” means named only in the other one, not absent from the study.

Nothing is left unaccounted for in this 455-domain table. The businesses named exactly twice fall between two definitions that were fixed before the data existed, and moving either threshold to collect them would be choosing a group after seeing the answers.

The result

P-values are two-sided Fisher exact tests on the 2×2 table for each signal. Generic arm; percentages are of the stores where the signal was observable. The two schema signals need a product page, and the scanner found one for only 10 of 55 cases and 46 of 181 controls here, and 13 of 52 cases and 33 of 131 controls in the context arm.

Context arm.

Pooled across the two arms — a case if it clears the threshold in either arm, a control if its higher arm total is one — 81 cases against 250 controls. This is the cut named in the amendment, and it is a union-of-arms classification rather than a conventional pooled estimate: a business enters the case group on its better arm, which is a selection effect worth naming.

The refutation condition was not met in the primary analysis. No signal differs by 20 points or more with the named group higher, in either arm or in the pooled cut. A separate runA-only check specified in the sealed plan did produce one such result; it is reported below because the rule was pre-specified, and its four-store case cell makes it uninterpretable. On the question the study set out to answer — do these four signals separate repeatedly named businesses from other businesses in the same named-business corpus that were named once — this study found no measurable evidence that they do.

Magento: the scanner classified 13 of the 455 successfully scanned domains as Magento — 2.9%, against the under-15% I specified before the analysis. Useful context, and worth stating plainly: the shops an assistant reaches for a specific home decor purchase are overwhelmingly not on the platform I build for. It is not an estimate of Magento’s share of AI-visible commerce generally; this is one category, three engines, two days.

Where I was wrong

The 15-point expectation failed. I wrote that no signal would separate the groups by more than 15 points. llms.txt separated them by 15.5, 17.4 and 15.3 across the three cuts, with p between .012 and .030. The refutation condition survived because it specified a direction and this went the other way — but the expectation as written did not, and I am not going to pretend the direction clause was foresight.

And in the runA-only cut, the refutation condition was nominally met. I kept the analysis as originally registered — one run, threshold three — so it could be checked against the sealed document. In that cut offers.availability differs by +22.5 points with cases higher, which is exactly the number I said would mean I was wrong.

It is not interpretable as evidence of an effect. The case group there is four stores with an observable product page. Two of four against eleven of forty, p = .570. I am reporting it because it is in the document I sealed, and a refutation condition you quietly drop when it fires on n=4 is not a refutation condition. But no one should read anything into it, including me.

The one number that went the other way

llms.txt is more common among the businesses engines name once than among those they name repeatedly. The direction is the same in all five cuts reported here — generic arm, context arm, pooled, runA-only and the threshold-two sensitivity check — and the three primary llms.txt comparisons produce nominal p-values below .05. These are four signals across five cuts with no multiplicity correction applied, so the p-values are worth reporting and not worth treating as confirmatory.

The obvious reading — publishing an llms.txt makes you less visible — is not supported by this analysis. Here is what I found when I went looking.

71% of the llms.txt files appear to come from the same template. The scanner records only whether the file exists; it keeps no page content, by policy. So for this exploratory check I separately fetched the 159 files it had flagged as present across everything scanned and read the first line of each. 113 matched the same generated-heading template, differing only in the brand name. Not byte-identical files. That points at a shared template or a platform-generated implementation rather than 113 shops each deciding to publish; I did not establish which generator produced it. (A repeat of the same check returned 113 against 112 the first time; one site answered differently between fetches, which is the size of the noise in this count.)

Narrowed to the 455 domains in the analysis universe, so the denominators match everything above: 146 serve an llms.txt, which is 32%, and 104 of those 146 — again 71% — are the template.

Strip the template and 42 of the 455 remain — 9%. On my own frame of 762 Magento stores the rate was 85 of 762, 11%. What looked like a large difference in adoption between two populations was almost entirely one piece of boilerplate. I was about to report 34% as adoption. This answers a different question from the case-control comparison above. The template finding explains most of the difference in overall llms.txt adoption between the two populations; it does not explain why controls in this sample were more likely than cases to serve the file.

Splitting by file kind, both halves lean the same way. Among the 58 generic-arm cases, 8 serve a templated file and 2 a non-templated one; among the 196 controls, 43 and 18. The non-templated files did not remove the direction — they too were more common among controls — but with only two non-templated cases, that half carries no weight on its own. Non-templated here means the heading did not match; I did not verify that anyone wrote those files by hand.

Splitting the effect this way destroys the power to interpret either half. So the honest position is: the association is there, it is consistent, its direction is the opposite of what anyone selling AEO would like, and this study cannot tell you why. One plausible hypothesis is that controls are disproportionately smaller or newer shops on hosted platforms where the file is generated by default, and cases are larger established retailers on older custom stacks. It is a hypothesis I did not test.

This section is exploratory. It was not in the sealed plan, the split was decided after seeing that the two population rates disagreed, and it is labelled as such here so nobody has to wonder.

What this does not show

Two of the four signals were only partially observable. JSON-LD and offers.availability need a product page, and the scanner found one on 10 of 55 cases and 46 of 181 controls in the generic arm. Those effective samples are too small to carry a conclusion either way, and the missingness may not be random: product-page discovery appears to work worst on the larger sites that dominate the case group. I do not read the schema rows as evidence for or against an association.

Losses are not random with respect to the outcome either. Businesses lost to bot mitigation — 403 and 429 — could also be more likely to disallow AI crawlers, which is one of the four signals. And 186 of 669 rows had no resolvable domain at all.

This is not a predictor-of-visibility study. Because every business in the analysed universe was named at least once, the comparison cannot say what makes a business enter an AI answer in the first place. It compares repeated mentions against one-off mentions among businesses already in the answers.

Crawler access is not the only route into an answer. An assistant can name a shop from a search index, a cached page, a product feed, a third-party listing or an earlier crawl. So the robots.txt signal measures whether one specific fetch path is open, not whether the shop can be mentioned at all.

Association, not causation. Nothing here randomises anything. A shop that publishes structured data and gets named may do both because someone there thinks about being found.

One category, and the first of three. Clothing and electronics accessories were agreed with connexion.me and were scheduled to run on 12 August. They had not arrived when this was written. Both will be published under the definitions above, whatever they show; if they do not materialise, that will be said here rather than left as silence.

One instrument, on days it was pinned. Two of the three engines record exact model ids. Perplexity’s sonar cannot be versioned — its models endpoint returns 404 and a versioned name is refused — so if the model behind it were replaced, nothing in either party’s records would show it.

A note on the other side of this

I have been checking connexion.me’s numbers for two weeks and publishing the result either way. In that time they have sent me, unprompted and before I could find any of it: a fold rule that behaved differently in the two arms; two duplicate rows I had missed and nine more the domain column caught; a set of published method examples that had been typed by hand rather than produced by the rule they illustrated; a repeat run that cut their own headline from 181 businesses against 143 to 187 against 133, with 109 of the original 181 never named again; a correction to that last number, because 109 was the count named exactly once and the right figure is 110; and, while this was being written, notice that one ranking file in a package I already hold had been superseded on their own site, along with a defect in it that neither of us had caught.

I have no way to audit an instrument I did not build. What I can report is that every time something in their data was about to reach a public result under my name, the correction came from them first.

Reproducing it

The sealed analysis plan and both amendments, with their hashes — 9b4ccf12629e…, 45cd9e03350b…, 16cb2a3b02b7…; the marketplace exclusion list and the dropped-domain list, both fixed before any signal was read; the per-business analysis rows; the scanner. All published. connexion.me’s roster, corpora and normalisation config are theirs, and go up on their site after this.

The roster domains were scanned between 10 and 15 August 2026. The scanner identifies itself, obeys robots.txt, keeps no page content and can be blocked in two lines — the crawler policy has the details, including why no individual business is named in this article.

Earlier instalments: the baseline scan, 462 stores in July and 762 in August, and 376 stores measured twice.