# Robots.txt AEO for Magento 2  -  module reference

> Manage AI crawler rules in Magento 2 robots.txt and keep your own rules. Bot list, Inject and Replace modes, content signals, bot IP checks.

Source: https://angeo.dev/docs/modules/robots-txt-aeo/
Updated: 2026-10-17

Module reference

# Robots.txt AEO for Magento 2  -  module reference

Covers `angeo/module-robots-txt-aeo` 4.0.1 - Checked 16 October 2026 - MIT licensed

**Robots.txt AEO** adds rules for AI crawlers to your Magento 2 `robots.txt`. It does this at render time and keeps your own rules as they are. You tick which bots may read the store in the admin panel.

Since 4.0.0 it can also **check** if a crawler is real. A `User-Agent: GPTBot` header is one line anyone can send. A request signed by the vendor, or an address inside the vendor's published IP range, is proof.

Package`angeo/module-robots-txt-aeo`Latest version4.0.1Magento2.4.xPHP8.2 - 8.5LicenseMIT, freeLinks[GitHub](https://github.com/angeo-dev/module-robots-txt-aeo) - [Packagist](https://packagist.org/packages/angeo/module-robots-txt-aeo) - [Changelog](https://github.com/angeo-dev/module-robots-txt-aeo/blob/main/CHANGELOG.md) - [Module page](https://angeo.dev/modules/robots-txt-aeo/)

**On this page**

1. What it does
2. Requirements
3. Installation
4. How the block is added
5. Content-use signals and licensing
6. Command line
7. Checking if a bot is real
8. Configuration reference
9. FAQ
10. Related

## What it does

- Adds a managed block of AI bot rules to `robots.txt`. Nothing is written to the database or the file system.
- Ships a catalogue of AI bots: OAI-SearchBot, GPTBot, ChatGPT-User, OAI-AdsBot, PerplexityBot, Perplexity-User, Google-Extended, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, Applebot, Applebot-Extended, cohere-ai, Amazonbot, Meta-ExternalAgent, meta-externalfetcher, CCBot, Bytespider, MistralAI-User and DuckAssistBot.
- Lets you set `Allow`, `Disallow` and `Crawl-delay` per bot.
- Adds `Sitemap:` lines, from Magento's sitemap or your own list.
- Can write content-use signals (IETF `Content-Usage`, Cloudflare `Content-Signal`) and an RSL `License:` line.
- Verifies bots by IP range and by RFC 9421 request signature (Web Bot Auth). It only reports. It never blocks.

## Requirements

| Magento | 2.4.x (Open Source, Mage-OS or Adobe Commerce)  -  `magento/framework ^103.0` |
| PHP | 8.2 - 8.5  -  `~8.2.0||~8.3.0||~8.4.0||~8.5.0` |
| PHP extensions | `json`, `sodium` |

PHP 8.1 was dropped in 4.0.0. If you are on PHP 8.1, stay on 3.0.x. The `sodium` extension ships with PHP and is needed for signature checks.

## Installation

```
composer require angeo/module-robots-txt-aeo
bin/magento module:enable Angeo_RobotsTxtAeo
bin/magento setup:upgrade
bin/magento setup:di:compile                # production mode only
bin/magento setup:static-content:deploy     # production mode only
bin/magento cache:flush
```

The module is on by default. The mainstream AI bots are allowed. Lower-traffic bots and the six tokens added in 4.0.0 are in the catalogue but switched off.

## How the block is added

A plugin on `Magento\Robots\Model\Robots::getData()` adds the AI block when `robots.txt` is rendered. Your own text under *Content -> Design -> Configuration -> Search Engine Robots* stays untouched.

### Inject mode (default, recommended)

The AI block goes before your existing rules. Only lines the module owns are replaced. Your comments, spacing and rule order stay byte for byte. Running it again replaces the block. It never stacks.

```
# Angeo AEO  -  AI Crawler Rules
User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /
Disallow: /admin/
# End Angeo AEO block

User-agent: *
Disallow: /checkout/
... your own rules, unchanged
```

### Replace mode

The module builds the whole file and keeps your custom `Disallow` rules from the wildcard group. Use it only if the module should own the full file.

Since 4.0.0, Replace mode will not open a closed site. If your file has `User-agent: *` with `Disallow: /` (common on staging), it is served unchanged and the admin says why.

## Content-use signals and licensing

These settings are all off by default:

- **IETF `Content-Usage`**  -  an aipref statement such as `train-ai=n` (no AI training).
- **Cloudflare `Content-Signal`**  -  separate yes/no values for `search`, `ai-train` and `ai-input`.
- **RSL `License:`**  -  a link to your RSL license XML file.

By default, signals go into the wildcard group, as Cloudflare's managed `robots.txt` does. Set the placement to *per bot* for the 3.x layout.

## Command line

```
# Show what would be added, without applying it
bin/magento angeo:robots:preview --store=1

# Fetch the live robots.txt and check the enabled bot rules are there
bin/magento angeo:robots:validate --store=1

# Is this address inside a published AI crawler range?
bin/magento angeo:robots:verify-bot-ip 203.0.113.10 --bot=GPTBot

# Was this request really signed by the vendor?
bin/magento angeo:robots:verify-bot-request --headers-file=/tmp/headers.txt \
    --authority=shop.example --path=/product.html --ip=203.0.113.10
```

| Command | What it does |

| `bin/magento angeo:robots:preview` | Preview the final robots.txt output after AI rule injection (no changes made) |
| `bin/magento angeo:robots:validate` | Validate that AI crawler rules are present in the live robots.txt |
| `bin/magento angeo:robots:verify-bot-ip` | Verify an IP address against vendor-published AI bot IP ranges |
| `bin/magento angeo:robots:verify-bot-request` | Verify a captured request against its Web Bot Auth signature |

`validate` exits with a non-zero code when expected rules are missing. Use it as a smoke test after deploy.

## Checking if a bot is real

A check returns one of four states:

| State | Meaning |

| `verified` | The signature is valid, or the address is in the vendor's range. |
| `failed` | It is not. Treat the request as fake. |
| `unknown` | No decision: the key directory was not reachable, or no IP was given. This is **not** the same as failed. |
| `unsupported` | The vendor publishes no way to check this bot. |

What vendors publish today:

| Vendor | IP ranges | Signed requests |

| OpenAI | Yes, three feeds | Yes, `https://chatgpt.com` |
| Anthropic | Yes, one shared feed for ClaudeBot, Claude-User and Claude-SearchBot | No |
| Perplexity | Yes, one feed per bot | No |
| Google, Apple, Meta, ByteDance, Mistral, DuckDuckGo | No | No |

A match in Anthropic's shared feed proves the request came from Anthropic, but not which of its three bots sent it. The module says exactly that. The `Signature-Agent` header is never trusted alone. You can add new signing origins under **Bot Verification** when a vendor publishes one between releases.

Other modules can use the checks through `Angeo\RobotsTxtAeo\Api\BotVerificationInterface`, and read the effective file through `Api\RobotsStatusInterface`. [AEO Audit](https://angeo.dev/docs/modules/aeo-audit/) uses the second one, so it needs no HTTP request when both modules are installed.

## Configuration reference

All settings are under **Stores -> Configuration -> Angeo -> Robots.txt AEO**. The table shows each field, its config path (for `bin/magento config:set`) and its default value.

General (3)

| Setting | Config path | Default |

| Enable AI Bot Rules | `angeo_robots_txt_aeo/general/enabled` | Yes |
| Injection Mode | `angeo_robots_txt_aeo/general/mode` | `inject` |
| Custom robots.txt Content (Replace mode) | `angeo_robots_txt_aeo/general/custom_content` |  -  |

AI Crawlers (21)

| Setting | Config path | Default |

| ★ OAI-SearchBot (ChatGPT live search) | `angeo_robots_txt_aeo/bots/oai_searchbot` | Yes |
| ★ GPTBot (OpenAI training crawler) | `angeo_robots_txt_aeo/bots/gptbot` | Yes |
| ChatGPT-User (ChatGPT browsing) | `angeo_robots_txt_aeo/bots/chatgpt_user` | Yes |
| OAI-AdsBot (ChatGPT ads validation) | `angeo_robots_txt_aeo/bots/oai_adsbot` | No |
| PerplexityBot (Perplexity index crawler) | `angeo_robots_txt_aeo/bots/perplexitybot` | Yes |
| Perplexity-User (Perplexity real-time fetch) | `angeo_robots_txt_aeo/bots/perplexity_user` | Yes |
| ★ Google-Extended (Gemini / AI Overviews) | `angeo_robots_txt_aeo/bots/google_extended` | Yes |
| ClaudeBot (Anthropic Claude) | `angeo_robots_txt_aeo/bots/claudebot` | Yes |
| anthropic-ai  -  DEPRECATED by Anthropic | `angeo_robots_txt_aeo/bots/anthropic_ai` | No |
| Claude-User (Claude browsing agent) | `angeo_robots_txt_aeo/bots/claude_user` | Yes |
| Claude-SearchBot (Claude search index) | `angeo_robots_txt_aeo/bots/claude_searchbot` | Yes |
| Applebot (Apple Intelligence / Siri) | `angeo_robots_txt_aeo/bots/applebot` | Yes |
| cohere-ai (Cohere training crawler) | `angeo_robots_txt_aeo/bots/cohere_ai` | No |
| Amazonbot (Amazon Alexa AI) | `angeo_robots_txt_aeo/bots/amazonbot` | No |
| Meta-ExternalAgent (Meta AI) | `angeo_robots_txt_aeo/bots/meta_external_agent` | No |
| Applebot-Extended (Apple model training) | `angeo_robots_txt_aeo/bots/applebot_extended` | No |
| meta-externalfetcher (Meta AI user fetch) | `angeo_robots_txt_aeo/bots/meta_external_fetcher` | No |
| CCBot (Common Crawl) | `angeo_robots_txt_aeo/bots/ccbot` | No |
| Bytespider (ByteDance training) | `angeo_robots_txt_aeo/bots/bytespider` | No |
| MistralAI-User (Le Chat user fetch) | `angeo_robots_txt_aeo/bots/mistralai_user` | No |
| DuckAssistBot (DuckDuckGo AI answers) | `angeo_robots_txt_aeo/bots/duckassistbot` | No |

AI Crawler Path Overrides (18)

| Setting | Config path | Default |

| Allow paths | `angeo_robots_txt_aeo/bot_overrides/oai_searchbot/allow` |  -  |
| Disallow paths | `angeo_robots_txt_aeo/bot_overrides/oai_searchbot/disallow` |  -  |
| Crawl-delay (seconds) | `angeo_robots_txt_aeo/bot_overrides/oai_searchbot/crawl_delay` |  -  |
| Allow paths | `angeo_robots_txt_aeo/bot_overrides/gptbot/allow` |  -  |
| Disallow paths | `angeo_robots_txt_aeo/bot_overrides/gptbot/disallow` |  -  |
| Allow paths | `angeo_robots_txt_aeo/bot_overrides/chatgpt_user/allow` |  -  |
| Disallow paths | `angeo_robots_txt_aeo/bot_overrides/chatgpt_user/disallow` |  -  |
| Crawl-delay (seconds) | `angeo_robots_txt_aeo/bot_overrides/chatgpt_user/crawl_delay` |  -  |
| Allow paths | `angeo_robots_txt_aeo/bot_overrides/perplexitybot/allow` |  -  |
| Disallow paths | `angeo_robots_txt_aeo/bot_overrides/perplexitybot/disallow` |  -  |
| Crawl-delay (seconds) | `angeo_robots_txt_aeo/bot_overrides/perplexitybot/crawl_delay` |  -  |
| Allow paths | `angeo_robots_txt_aeo/bot_overrides/google_extended/allow` |  -  |
| Disallow paths | `angeo_robots_txt_aeo/bot_overrides/google_extended/disallow` |  -  |
| Allow paths | `angeo_robots_txt_aeo/bot_overrides/claudebot/allow` |  -  |
| Disallow paths | `angeo_robots_txt_aeo/bot_overrides/claudebot/disallow` |  -  |
| Allow paths | `angeo_robots_txt_aeo/bot_overrides/applebot/allow` |  -  |
| Disallow paths | `angeo_robots_txt_aeo/bot_overrides/applebot/disallow` |  -  |
| Crawl-delay (seconds) | `angeo_robots_txt_aeo/bot_overrides/applebot/crawl_delay` |  -  |

Sitemap Directive (2)

| Setting | Config path | Default |

| Sitemap source | `angeo_robots_txt_aeo/sitemap/mode` | `auto` |
| Custom sitemap URLs | `angeo_robots_txt_aeo/sitemap/custom_urls` |  -  |

Content-Usage Signals (8)

| Setting | Config path | Default |

| Emit IETF Content-Usage directive | `angeo_robots_txt_aeo/content_signals/ietf_enabled` | No |
| Content-Usage preference | `angeo_robots_txt_aeo/content_signals/ietf_preference` | `train-ai=n` |
| Emit Cloudflare Content-Signal directive | `angeo_robots_txt_aeo/content_signals/cloudflare_enabled` | No |
| search  -  build a search index, show links/snippets | `angeo_robots_txt_aeo/content_signals/cf_search` | `yes` |
| ai-train  -  train or fine-tune AI models | `angeo_robots_txt_aeo/content_signals/cf_ai_train` | `no` |
| ai-input  -  use as input to AI answers (RAG/grounding) | `angeo_robots_txt_aeo/content_signals/cf_ai_input` | `unset` |
| Where to write the signals | `angeo_robots_txt_aeo/content_signals/placement` | `wildcard` |
| Include the short policy explanation | `angeo_robots_txt_aeo/content_signals/policy_comment` | Yes |

Bot Verification (1)

| Setting | Config path | Default |

| Additional trusted signing origins | `angeo_robots_txt_aeo/verification/trusted_signature_agents` |  -  |

Content Licensing (RSL) (2)

| Setting | Config path | Default |

| Emit RSL License directive | `angeo_robots_txt_aeo/licensing/rsl_enabled` | No |
| RSL license file URL | `angeo_robots_txt_aeo/licensing/rsl_license_url` |  -  |

All settings work per store view. The **Live Preview** group shows the block that will be added.

## FAQ

### Will it overwrite my robots.txt?

No, not in Inject mode, which is the default. It adds its own block and keeps your rules, comments and order exactly as they are.

### Should I block GPTBot or ClaudeBot?

That is a licensing choice. These bots collect pages for training. Blocking them does not remove you from AI search. Search bots such as OAI-SearchBot, PerplexityBot and Claude-SearchBot are the ones that get you cited.

### Does it block fake bots?

No. It only reports if a bot is verified, failed, unknown or unsupported. Blocking belongs in your WAF or CDN.

### Why is my change not visible on the live site?

If Varnish, Fastly or another CDN caches `robots.txt`, purge it after you change the settings.

### Does it work with Hyvä?

Yes. `robots.txt` is built on the server, so the theme does not matter.

## Related

- [AEO Audit](https://angeo.dev/docs/modules/aeo-audit/)  -  score 20 AI-visibility signals from the command line
- [llms.txt](https://angeo.dev/docs/modules/llms-txt/)  -  generate llms.txt, llms.jsonl, Markdown mirrors and agents.md
- [AI Agent Activity](https://angeo.dev/docs/modules/agent-activity/)  -  report on AI agent calls, crawler visits, fake bots and agent-built orders
- [AI crawlers for Magento 2 (guide)](https://angeo.dev/docs/ai-crawlers/)
- [Spoofed AI crawler  -  definition](https://angeo.dev/spoofed-ai-crawler-definition/)
- [Web Bot Auth  -  definition](https://angeo.dev/web-bot-auth-definition/)
- [All module reference pages](https://angeo.dev/docs/modules/) - [Compatibility matrix](https://angeo.dev/docs/compatibility/)

This page was written from the `README.md`, `composer.json` and `etc/*.xml` files of [tag 4.0.1](https://github.com/angeo-dev/module-robots-txt-aeo/tree/4.0.1), read on 16 October 2026. Settings, commands and defaults come straight from that code. If a newer release is out, the README on GitHub wins.