AngeoBenchmarkBot is a low-rate research crawler for an open study of how Magento 2 stores are set up for AI assistants — the crawlers and answer engines behind ChatGPT, Claude, Perplexity and Google’s AI features. It is run by Ievgenii Gryshkun (angeo.dev), an independent Magento 2 solution architect based in the Netherlands.

In short. It reads a handful of publicly served pages, obeys robots.txt, waits at least two seconds between requests, never interacts with your store, keeps no page content, and publishes aggregate figures only. If you found it in your access logs, nothing is wrong.

It does not collect customer data, submit forms, create accounts, log in, add products to a cart, follow checkout links, or request admin paths. It does not build a mailing list, and nobody is contacted as a result of being crawled.

The exact User-Agent string, for grepping your logs:

AngeoBenchmarkBot/1.0 (+https://angeo.dev/magento-ai-visibility-statistics/; research scan; contact info@angeo.dev)

What it requests

Between two and six requests per site, spread over a few minutes. Then it leaves and does not return until the next round of the study.

Path What is measured
/ Whether the site runs Magento 2, from public markup traces. The title, meta description and html lang are read to classify the shop by what it sells.
/robots.txt Whether each of eight AI crawlers is allowed or disallowed: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended.
/llms.txt Whether the file exists, returns 200 and is not empty. The contents are not parsed, stored or analysed — only that boolean.
One product page, reached from a homepage link Two booleans from JSON-LD: whether a @type: Product node exists, and whether any of its offers carries an availability value. Prices, SKUs, brands and images are not read or recorded.

How it behaves

  • Reads robots.txt before anything else and obeys it, including Crawl-delay.
  • Waits at least two seconds between requests to the same host.
  • Honours Retry-After and backs off on 429 or 503, then gives up rather than pushing.
  • Reads at most the first few hundred kilobytes of a response.
  • Runs single-threaded per host, from one machine.

What is stored

One record per domain: the domain, its rank in the public source list, the homepage title, description and language, whether the site was reachable, the allow/disallow status of the eight crawlers above, the booleans in the table, the URL of the single product page tested, and a timestamp.

No page content is retained. HTML is parsed in memory and discarded; nothing is archived, cached or republished. No personal data is collected — the crawler reads shop pages, not customers, and records no IP addresses of anyone. Records are kept while the study runs and for twelve months after publication so the figures can be checked, then deleted. They are used for this benchmark only: not sold, not shared, not used for prospecting.

What gets published

Aggregate numbers only — for example, “X per cent of the stores measured publish JSON-LD product data”. No individual store is named, ranked, scored or identified in any publication without the written consent of its owner.

The sample frame comes from the public Tranco list. The list ID, the date, and any domains excluded from the frame are published with the results, so the sample can be reconstructed and the work checked independently.

How to keep it out

Add this to your robots.txt:

User-agent: AngeoBenchmarkBot
Disallow: /

It is read and obeyed on the next visit. You can also email info@angeo.dev with your domain and it will be excluded from the frame permanently — no reason needed, no reply expected. Exclusion requests are honoured before the results are compiled, not after.

Contact

Questions, complaints, or a request to see the stored record for your own domain: info@angeo.dev. A record is sent to whoever can demonstrate control of the domain, and deleted on request.

A User-Agent string is a claim, not proof — anything can send one. If something using this name is hitting you hard, or behaving differently from what is described above, please send me a log excerpt. It is not mine, and I would rather know.

Last updated: 8 August 2026.