> A low-rate research crawler measuring how Magento 2 stores are set up for AIassistants. What it requests, what is stored, and how to block it.

Source: https://angeo.dev/magento-ai-visibility-statistics/
Updated: 2026-09-25

# AngeoBenchmarkBot crawler policy

 **AngeoBenchmarkBot** is a low-rate research crawler for an open
 study of how Magento 2 stores are set up for AI assistants  -  the
 crawlers and answer engines behind ChatGPT, Claude, Perplexity and Google's
 AI features. It is run by Ievgenii Gryshkun
 ([angeo.dev](https://angeo.dev/)),
 an independent Magento 2 solution architect based in the Netherlands.

 **In short.** It reads a handful of publicly served pages,
 obeys `robots.txt`, waits at least two seconds between requests,
 never interacts with your store, keeps no page content, and publishes
 aggregate figures only. If you found it in your access logs, nothing is
 wrong.

 It does not collect customer data, submit forms, create accounts, log in,
 add products to a cart, follow checkout links, or request admin paths. It
 does not build a mailing list, and nobody is contacted as a result of being
 crawled.

 The exact User-Agent string, for grepping your logs:

```
AngeoBenchmarkBot/1.0 (+https://angeo.dev/magento-ai-visibility-statistics/; research scan; contact info@angeo.dev)
```

## What it requests

 Between two and six requests per site, spread over a few minutes. Then it
 leaves and does not return until the next round of the study.

| Path | What is measured |

| `/` | Whether the site runs Magento 2, from public markup traces. The title, meta description and `html lang` are read to classify the shop by what it sells. |

| `/robots.txt` | Whether each of eight AI crawlers is allowed or disallowed: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended. |

| `/llms.txt` | Whether the file exists, returns 200 and is not empty. The contents are not parsed, stored or analysed  -  only that boolean. |

| One product page, reached from a homepage link | Two booleans from JSON-LD: whether a `@type: Product` node exists, and whether any of its `offers` carries an `availability` value. Prices, SKUs, brands and images are not read or recorded. |

## How it behaves

- Reads `robots.txt` before anything else and obeys it, including `Crawl-delay`.

- Waits at least two seconds between requests to the same host.

- Honours `Retry-After` and backs off on `429` or `503`, then gives up rather than pushing.

- Reads at most the first few hundred kilobytes of a response.

- Runs single-threaded per host, from one machine.

## What is stored

 One record per domain: the domain, its rank in the public source list, the
 homepage title, description and language, whether the site was reachable,
 the allow/disallow status of the eight crawlers above, the booleans in the
 table, the URL of the single product page tested, and a timestamp.

 **No page content is retained.** HTML is parsed in memory and
 discarded; nothing is archived, cached or republished. No personal data is
 collected  -  the crawler reads shop pages, not customers, and records no IP
 addresses of anyone. Records are kept while the study runs and for twelve
 months after publication so the figures can be checked, then deleted. They
 are used for this benchmark only: not sold, not shared, not used for
 prospecting.

## What gets published

 Aggregate numbers only  -  for example, *"X per cent of the stores measured
 publish JSON-LD product data"*. No individual store is named, ranked,
 scored or identified in any publication without the written consent of its
 owner.

 The sample frame comes from the public
 [Tranco](https://tranco-list.eu/)
 list. The list ID, the date, and any domains excluded from the frame are
 published with the results, so the sample can be reconstructed and the work
 checked independently.

## How to keep it out

 Add this to your `robots.txt`:

```
User-agent: AngeoBenchmarkBot
Disallow: /
```

 It is read and obeyed on the next visit. You can also email
 [info@angeo.dev](mailto:info@angeo.dev)
 with your domain and it will be excluded from the frame permanently  -  no
 reason needed, no reply expected. Exclusion requests are honoured before the
 results are compiled, not after.

## Contact

 Questions, complaints, or a request to see the stored record for your own
 domain:
 [info@angeo.dev](mailto:info@angeo.dev).
 A record is sent to whoever can demonstrate control of the domain, and
 deleted on request.

 A User-Agent string is a claim, not proof  -  anything can send one. If
 something using this name is hitting you hard, or behaving differently from
 what is described above, please send me a log excerpt. It is not mine, and I
 would rather know.

 Last updated: 8 August 2026.

## More on this topic

- [462 Magento Stores Scanned for AI Readability](https://angeo.dev/aeo-scan-case-study/)
- [376 Magento stores, measured twice: 94% did not change](https://angeo.dev/magento-ai-signals-two-scans/)
- [458 Shops Named by AI: Four Signals Did Not Predict It](https://angeo.dev/ai-readability-and-being-named/)
- [Magento AI Visibility from 20% to 86%: Case Study](https://angeo.dev/magento-ai-visibility-case-study-20-to-86/)
- [Perplexity vs ChatGPT vs Gemini: AI Product Discovery](https://angeo.dev/perplexity-vs-chatgpt-vs-gemini-how-each-ai-discovers-your-products/)
- [What the Magento Ecosystem Has Actually Built for AI Search](https://angeo.dev/magento-aeo-ecosystem-what-has-been-built/)