# robots.txt for AI Crawlers

> Manage rules for 13 AI crawlers in Magento 2 robots.txt  -  OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot and more. RFC 9309 safe editing. Free.

Source: https://angeo.dev/modules/robots-txt-aeo/
Updated: 2026-10-17

02 - Access - Open source

# AI Crawler Rules in robots.txt for Magento 2

 angeo/module-robots-txt-aeo

 Manages AI crawler directives in robots.txt from the Magento admin  -  OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, ClaudeBot, anthropic-ai, Claude-User, Applebot, cohere-ai, Amazonbot and Meta-ExternalAgent  -  without destroying the rules you already have.

 v4.0.0RFC 9309RSL 1.0MIT

 # install
$ composer require angeo/module-robots-txt-aeo
$ bin/magento setup:upgrade
# confirm a crawler is who it claims to be
$ bin/magento angeo:robots:verify-bot-ip 12.34.56.78

 Free and MIT-licensed. No licence key, no account, no telemetry.

## What it does

 Four things worth knowing before you install.

### Lossless round-trip parsing

Your existing robots.txt is parsed to RFC 9309 and written back with comments, ordering and unknown directives intact. The module edits its own block and leaves everything else byte-identical.

### A current bot catalogue

Search crawlers, training crawlers and user-triggered fetchers are listed separately, because they are separate decisions. Deprecated agent names are flagged rather than silently kept.

### Usage terms, not just access

Optional IETF Content-Usage directives and an RSL 1.0 licence block state the terms under which content may be used, alongside the plain allow and disallow rules.

### Verify who is actually crawling

`angeo:robots:verify-bot-ip` checks whether a request claiming to be a named AI crawler came from that operator's published address ranges.

 **On Adobe Commerce Cloud, purge Fastly after any change.** robots.txt is cached at the edge, and AI crawlers will keep reading the old file  -  which is one of the most common reasons a fix appears not to work.

## Requirements

| Magento | Open Source / Adobe Commerce 2.4.x |

| PHP | 8.2+ |

| Standard | RFC 9309 |

| Scope | Per website / store view |

| Licence | MIT |

## Questions

### Blocking GPTBot removes me from ChatGPT, right?

No  -  and this is the single most common misconfiguration. GPTBot is the training crawler. OAI-SearchBot is what serves ChatGPT search answers. Blocking the first does not remove you from search; blocking the second does.

### Will it overwrite the robots.txt rules I already have?

No. Existing directives are preserved through a lossless parse and rewrite. The module only owns the block it generates.

### Can rules differ per store view?

Yes. Directives are scoped, which matters when store views sit on different domains or serve different markets.

### Does allowing Google-Extended affect my Google rankings?

No. Google-Extended governs use in AI experiences and is separate from Googlebot and from regular search ranking.

## Works with

 Each module is independent, but these three are the ones most often installed alongside it.

 [angeo/module-aeo-audit](https://angeo.dev/modules/aeo-audit/)
 [angeo/module-llms-txt](https://angeo.dev/modules/llms-txt/)
 [angeo/module-ucp](https://angeo.dev/modules/ucp/)

## Not sure whether you need this one?

 Run the free scanner on your domain, or install the CLI audit. Both report which signals your store currently fails, so you install only what closes a real gap.

 [Scan my store](https://angeo.dev/ai-magento-audit/)
 [All modules](https://angeo.dev/modules/)

## More on this topic

- [AI Crawlers for Magento 2: GPTBot, OAI-SearchBot & More](https://angeo.dev/docs/ai-crawlers/)
- [GPTBot vs OAI-SearchBot - Definition and Difference](https://angeo.dev/gptbot-vs-oai-searchbot/)
- [How to Control AI Overviews and AI Mode on Magento](https://angeo.dev/control-ai-overviews-magento/)
- [robots.txt for ChatGPT, Claude & Gemini in Magento 2](https://angeo.dev/magento-2-robots-txt-chatgpt-gemini-ai-bots/)
- [Content Signals and RSL - Definition](https://angeo.dev/content-signals-definition/)
- [Web Bot Auth - Definition and What It Means for Stores](https://angeo.dev/web-bot-auth-definition/)
- [Technical reference  -  every setting, command and default](https://angeo.dev/docs/modules/robots-txt-aeo/)

- [Spoofed AI crawler  -  definition](https://angeo.dev/spoofed-ai-crawler-definition/)

- [AI Agent Activity  -  module](https://angeo.dev/modules/agent-activity/)