02 · Access · Open source
AI Crawler Rules in robots.txt for Magento 2
angeo/module-robots-txt-aeo
Manages AI crawler directives in robots.txt from the Magento admin — OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, ClaudeBot, anthropic-ai, Claude-User, Applebot, cohere-ai, Amazonbot and Meta-ExternalAgent — without destroying the rules you already have.
Free and MIT-licensed. No licence key, no account, no telemetry.
What it does
Four things worth knowing before you install.
Lossless round-trip parsing
Your existing robots.txt is parsed to RFC 9309 and written back with comments, ordering and unknown directives intact. The module edits its own block and leaves everything else byte-identical.
A current bot catalogue
Search crawlers, training crawlers and user-triggered fetchers are listed separately, because they are separate decisions. Deprecated agent names are flagged rather than silently kept.
Usage terms, not just access
Optional IETF Content-Usage directives and an RSL 1.0 licence block state the terms under which content may be used, alongside the plain allow and disallow rules.
Verify who is actually crawling
angeo:robots:verify-bot-ip checks whether a request claiming to be a named AI crawler came from that operator’s published address ranges.
On Adobe Commerce Cloud, purge Fastly after any change. robots.txt is cached at the edge, and AI crawlers will keep reading the old file — which is one of the most common reasons a fix appears not to work.
Requirements
| Magento | Open Source / Adobe Commerce 2.4.x |
| PHP | 8.2+ |
| Standard | RFC 9309 |
| Scope | Per website / store view |
| Licence | MIT |
Questions
Blocking GPTBot removes me from ChatGPT, right?
No — and this is the single most common misconfiguration. GPTBot is the training crawler. OAI-SearchBot is what serves ChatGPT search answers. Blocking the first does not remove you from search; blocking the second does.
Will it overwrite the robots.txt rules I already have?
No. Existing directives are preserved through a lossless parse and rewrite. The module only owns the block it generates.
Can rules differ per store view?
Yes. Directives are scoped, which matters when store views sit on different domains or serve different markets.
Does allowing Google-Extended affect my Google rankings?
No. Google-Extended governs use in AI experiences and is separate from Googlebot and from regular search ranking.
Works with
Each module is independent, but these three are the ones most often installed alongside it.
Not sure whether you need this one?
Run the free scanner on your domain, or install the CLI audit. Both report which signals your store currently fails, so you install only what closes a real gap.