AI Crawlers and Magento: Which Bots to Allow, Which to Block, and Why

Should I block AI crawlers on my Magento store?

For a commerce store, generally no — at least not the search crawlers. Blocking them removes you from the answers where product discovery increasingly happens, and that visibility is hard to recover later. Blocking training crawlers is a defensible separate choice if you have proprietary content you do not want memorised.

Which AI bots should I allow to be cited in ChatGPT, Claude, and Perplexity?

Allow the search and user-facing agents for each vendor — these are what make you eligible to appear in answers. Vendors document their agents separately from their training crawlers, so consult current vendor documentation for exact names before writing rules, then implement them following the Magento robots.txt walkthrough.

Can I allow AI search but block AI training?

Yes, and this is a common enterprise pattern. Training and search agents are distinct user-agents with independent directives, so you can stay citable in AI answers while opting out of contributing to model training. The trade-off: staying out of training data can reduce long-term familiarity with your brand inside the models themselves.

Does blocking GPTBot remove me from ChatGPT?

Not by itself — the training crawler and the search crawler are different agents. Blocking only the training crawler leaves you eligible for ChatGPT’s search-based citations, provided the search agent is still allowed. Blocking both removes you from both.

Do AI crawlers actually respect robots.txt?

The major documented crawlers generally do, and you can verify it from your own server logs. But compliance is opt-in: some crawlers have been documented ignoring robots.txt, and user-directed fetches are treated by some providers as user actions rather than crawling, so robots.txt may not apply as site owners expect. Treat the file as a norm, not a guarantee.

How do I verify that a crawler is really who it claims to be?

Do not trust the user-agent string — anyone can send any header. Verify against the vendor’s published IP ranges where available, or fall back to a reverse DNS lookup confirmed by a forward lookup. This matters when you are deciding whether to serve or rate-limit traffic based on identity.

What happens to redirects and redirect chains?

Search-time agents have low tolerance for extra hops — an unnecessary redirect can be enough for a page not to be used in a generated answer. This applies with particular force to llms.txt: if it is served via redirect rather than directly at the root, some crawlers retry the root instead of following, making the file effectively inoperative.

Should I block anything at all?

Two categories are worth restricting regardless of your AI policy. Keep admin, login, cart, checkout, and account paths disallowed for every crawler — they have no discovery value and should not be fetched. And crawlers documented as non-compliant are better handled at the server or CDN layer, since a robots.txt rule they ignore achieves nothing.

Does allowing AI crawlers slow down my store?

It adds crawl traffic, which on a large catalogue is worth monitoring. Treat it as a cost-benefit question: measure crawler activity in your logs against citations and referral traffic, and restrict any agent that generates load with no discoverable benefit.

How often should I review my crawler rules?

At least quarterly. Agents get added, split into separate crawlers, and deprecated — a robots.txt copied from an older example can silently exclude you from surfaces that did not exist when it was written.

Is robots.txt enough on its own for AI visibility?

No. Crawler access is necessary but not sufficient: it lets AI systems reach your pages, while llms.txt, complete product schema, and a product feed determine what they can actually do with them. See the Magento 2 AEO guide for the full set, or compare implementation routes in AEO options for Magento.