Robots.txt AEO for Magento 2 — module reference
Robots.txt AEO adds rules for AI crawlers to your Magento 2 robots.txt. It does this at render time and keeps your own rules as they are. You tick which bots may read the store in the admin panel.
Since 4.0.0 it can also check if a crawler is real. A User-Agent: GPTBot header is one line anyone can send. A request signed by the vendor, or an address inside the vendor’s published IP range, is proof.
What it does
- Adds a managed block of AI bot rules to
robots.txt. Nothing is written to the database or the file system. - Ships a catalogue of AI bots: OAI-SearchBot, GPTBot, ChatGPT-User, OAI-AdsBot, PerplexityBot, Perplexity-User, Google-Extended, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, Applebot, Applebot-Extended, cohere-ai, Amazonbot, Meta-ExternalAgent, meta-externalfetcher, CCBot, Bytespider, MistralAI-User and DuckAssistBot.
- Lets you set
Allow,DisallowandCrawl-delayper bot. - Adds
Sitemap:lines, from Magento’s sitemap or your own list. - Can write content-use signals (IETF
Content-Usage, CloudflareContent-Signal) and an RSLLicense:line. - Verifies bots by IP range and by RFC 9421 request signature (Web Bot Auth). It only reports. It never blocks.
Requirements
| Magento | 2.4.x (Open Source, Mage-OS or Adobe Commerce) — magento/framework ^103.0 |
|---|---|
| PHP | 8.2 – 8.5 — ~8.2.0||~8.3.0||~8.4.0||~8.5.0 |
| PHP extensions | json, sodium |
PHP 8.1 was dropped in 4.0.0. If you are on PHP 8.1, stay on 3.0.x. The sodium extension ships with PHP and is needed for signature checks.
Installation
composer require angeo/module-robots-txt-aeo
bin/magento module:enable Angeo_RobotsTxtAeo
bin/magento setup:upgrade
bin/magento setup:di:compile # production mode only
bin/magento setup:static-content:deploy # production mode only
bin/magento cache:flushThe module is on by default. The mainstream AI bots are allowed. Lower-traffic bots and the six tokens added in 4.0.0 are in the catalogue but switched off.
How the block is added
A plugin on Magento\Robots\Model\Robots::getData() adds the AI block when robots.txt is rendered. Your own text under Content → Design → Configuration → Search Engine Robots stays untouched.
Inject mode (default, recommended)
The AI block goes before your existing rules. Only lines the module owns are replaced. Your comments, spacing and rule order stay byte for byte. Running it again replaces the block. It never stacks.
# Angeo AEO — AI Crawler Rules
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
Disallow: /admin/
# End Angeo AEO block
User-agent: *
Disallow: /checkout/
... your own rules, unchangedReplace mode
The module builds the whole file and keeps your custom Disallow rules from the wildcard group. Use it only if the module should own the full file.
Since 4.0.0, Replace mode will not open a closed site. If your file has User-agent: * with Disallow: / (common on staging), it is served unchanged and the admin says why.
Content-use signals and licensing
These settings are all off by default:
- **IETF
Content-Usage** — an aipref statement such astrain-ai=n(no AI training). - **Cloudflare
Content-Signal** — separate yes/no values forsearch,ai-trainandai-input. - **RSL
License:** — a link to your RSL license XML file.
By default, signals go into the wildcard group, as Cloudflare’s managed robots.txt does. Set the placement to per bot for the 3.x layout.
Command line
# Show what would be added, without applying it
bin/magento angeo:robots:preview --store=1
# Fetch the live robots.txt and check the enabled bot rules are there
bin/magento angeo:robots:validate --store=1
# Is this address inside a published AI crawler range?
bin/magento angeo:robots:verify-bot-ip 203.0.113.10 --bot=GPTBot
# Was this request really signed by the vendor?
bin/magento angeo:robots:verify-bot-request --headers-file=/tmp/headers.txt \
--authority=shop.example --path=/product.html --ip=203.0.113.10| Command | What it does |
|---|---|
bin/magento angeo:robots:preview | Preview the final robots.txt output after AI rule injection (no changes made) |
bin/magento angeo:robots:validate | Validate that AI crawler rules are present in the live robots.txt |
bin/magento angeo:robots:verify-bot-ip | Verify an IP address against vendor-published AI bot IP ranges |
bin/magento angeo:robots:verify-bot-request | Verify a captured request against its Web Bot Auth signature |
validate exits with a non-zero code when expected rules are missing. Use it as a smoke test after deploy.
Checking if a bot is real
A check returns one of four states:
| State | Meaning |
|---|---|
verified | The signature is valid, or the address is in the vendor’s range. |
failed | It is not. Treat the request as fake. |
unknown | No decision: the key directory was not reachable, or no IP was given. This is not the same as failed. |
unsupported | The vendor publishes no way to check this bot. |
What vendors publish today:
| Vendor | IP ranges | Signed requests |
|---|---|---|
| OpenAI | Yes, three feeds | Yes, https://chatgpt.com |
| Anthropic | Yes, one shared feed for ClaudeBot, Claude-User and Claude-SearchBot | No |
| Perplexity | Yes, one feed per bot | No |
| Google, Apple, Meta, ByteDance, Mistral, DuckDuckGo | No | No |
A match in Anthropic’s shared feed proves the request came from Anthropic, but not which of its three bots sent it. The module says exactly that. The Signature-Agent header is never trusted alone. You can add new signing origins under Bot Verification when a vendor publishes one between releases.
Other modules can use the checks through Angeo\RobotsTxtAeo\Api\BotVerificationInterface, and read the effective file through Api\RobotsStatusInterface. AEO Audit uses the second one, so it needs no HTTP request when both modules are installed.
Configuration reference
All settings are under Stores → Configuration → Angeo → Robots.txt AEO. The table shows each field, its config path (for bin/magento config:set) and its default value.
General (3)
| Setting | Config path | Default |
|---|---|---|
| Enable AI Bot Rules | angeo_robots_txt_aeo/general/enabled | Yes |
| Injection Mode | angeo_robots_txt_aeo/general/mode | inject |
| Custom robots.txt Content (Replace mode) | angeo_robots_txt_aeo/general/custom_content | — |
AI Crawlers (21)
| Setting | Config path | Default |
|---|---|---|
| ★ OAI-SearchBot (ChatGPT live search) | angeo_robots_txt_aeo/bots/oai_searchbot | Yes |
| ★ GPTBot (OpenAI training crawler) | angeo_robots_txt_aeo/bots/gptbot | Yes |
| ChatGPT-User (ChatGPT browsing) | angeo_robots_txt_aeo/bots/chatgpt_user | Yes |
| OAI-AdsBot (ChatGPT ads validation) | angeo_robots_txt_aeo/bots/oai_adsbot | No |
| PerplexityBot (Perplexity index crawler) | angeo_robots_txt_aeo/bots/perplexitybot | Yes |
| Perplexity-User (Perplexity real-time fetch) | angeo_robots_txt_aeo/bots/perplexity_user | Yes |
| ★ Google-Extended (Gemini / AI Overviews) | angeo_robots_txt_aeo/bots/google_extended | Yes |
| ClaudeBot (Anthropic Claude) | angeo_robots_txt_aeo/bots/claudebot | Yes |
| anthropic-ai — DEPRECATED by Anthropic | angeo_robots_txt_aeo/bots/anthropic_ai | No |
| Claude-User (Claude browsing agent) | angeo_robots_txt_aeo/bots/claude_user | Yes |
| Claude-SearchBot (Claude search index) | angeo_robots_txt_aeo/bots/claude_searchbot | Yes |
| Applebot (Apple Intelligence / Siri) | angeo_robots_txt_aeo/bots/applebot | Yes |
| cohere-ai (Cohere training crawler) | angeo_robots_txt_aeo/bots/cohere_ai | No |
| Amazonbot (Amazon Alexa AI) | angeo_robots_txt_aeo/bots/amazonbot | No |
| Meta-ExternalAgent (Meta AI) | angeo_robots_txt_aeo/bots/meta_external_agent | No |
| Applebot-Extended (Apple model training) | angeo_robots_txt_aeo/bots/applebot_extended | No |
| meta-externalfetcher (Meta AI user fetch) | angeo_robots_txt_aeo/bots/meta_external_fetcher | No |
| CCBot (Common Crawl) | angeo_robots_txt_aeo/bots/ccbot | No |
| Bytespider (ByteDance training) | angeo_robots_txt_aeo/bots/bytespider | No |
| MistralAI-User (Le Chat user fetch) | angeo_robots_txt_aeo/bots/mistralai_user | No |
| DuckAssistBot (DuckDuckGo AI answers) | angeo_robots_txt_aeo/bots/duckassistbot | No |
AI Crawler Path Overrides (18)
| Setting | Config path | Default |
|---|---|---|
| Allow paths | angeo_robots_txt_aeo/bot_overrides/oai_searchbot/allow | — |
| Disallow paths | angeo_robots_txt_aeo/bot_overrides/oai_searchbot/disallow | — |
| Crawl-delay (seconds) | angeo_robots_txt_aeo/bot_overrides/oai_searchbot/crawl_delay | — |
| Allow paths | angeo_robots_txt_aeo/bot_overrides/gptbot/allow | — |
| Disallow paths | angeo_robots_txt_aeo/bot_overrides/gptbot/disallow | — |
| Allow paths | angeo_robots_txt_aeo/bot_overrides/chatgpt_user/allow | — |
| Disallow paths | angeo_robots_txt_aeo/bot_overrides/chatgpt_user/disallow | — |
| Crawl-delay (seconds) | angeo_robots_txt_aeo/bot_overrides/chatgpt_user/crawl_delay | — |
| Allow paths | angeo_robots_txt_aeo/bot_overrides/perplexitybot/allow | — |
| Disallow paths | angeo_robots_txt_aeo/bot_overrides/perplexitybot/disallow | — |
| Crawl-delay (seconds) | angeo_robots_txt_aeo/bot_overrides/perplexitybot/crawl_delay | — |
| Allow paths | angeo_robots_txt_aeo/bot_overrides/google_extended/allow | — |
| Disallow paths | angeo_robots_txt_aeo/bot_overrides/google_extended/disallow | — |
| Allow paths | angeo_robots_txt_aeo/bot_overrides/claudebot/allow | — |
| Disallow paths | angeo_robots_txt_aeo/bot_overrides/claudebot/disallow | — |
| Allow paths | angeo_robots_txt_aeo/bot_overrides/applebot/allow | — |
| Disallow paths | angeo_robots_txt_aeo/bot_overrides/applebot/disallow | — |
| Crawl-delay (seconds) | angeo_robots_txt_aeo/bot_overrides/applebot/crawl_delay | — |
Sitemap Directive (2)
| Setting | Config path | Default |
|---|---|---|
| Sitemap source | angeo_robots_txt_aeo/sitemap/mode | auto |
| Custom sitemap URLs | angeo_robots_txt_aeo/sitemap/custom_urls | — |
Content-Usage Signals (8)
| Setting | Config path | Default |
|---|---|---|
| Emit IETF Content-Usage directive | angeo_robots_txt_aeo/content_signals/ietf_enabled | No |
| Content-Usage preference | angeo_robots_txt_aeo/content_signals/ietf_preference | train-ai=n |
| Emit Cloudflare Content-Signal directive | angeo_robots_txt_aeo/content_signals/cloudflare_enabled | No |
| search — build a search index, show links/snippets | angeo_robots_txt_aeo/content_signals/cf_search | yes |
| ai-train — train or fine-tune AI models | angeo_robots_txt_aeo/content_signals/cf_ai_train | no |
| ai-input — use as input to AI answers (RAG/grounding) | angeo_robots_txt_aeo/content_signals/cf_ai_input | unset |
| Where to write the signals | angeo_robots_txt_aeo/content_signals/placement | wildcard |
| Include the short policy explanation | angeo_robots_txt_aeo/content_signals/policy_comment | Yes |
Bot Verification (1)
| Setting | Config path | Default |
|---|---|---|
| Additional trusted signing origins | angeo_robots_txt_aeo/verification/trusted_signature_agents | — |
Content Licensing (RSL) (2)
| Setting | Config path | Default |
|---|---|---|
| Emit RSL License directive | angeo_robots_txt_aeo/licensing/rsl_enabled | No |
| RSL license file URL | angeo_robots_txt_aeo/licensing/rsl_license_url | — |
All settings work per store view. The Live Preview group shows the block that will be added.
FAQ
Will it overwrite my robots.txt?
No, not in Inject mode, which is the default. It adds its own block and keeps your rules, comments and order exactly as they are.
Should I block GPTBot or ClaudeBot?
That is a licensing choice. These bots collect pages for training. Blocking them does not remove you from AI search. Search bots such as OAI-SearchBot, PerplexityBot and Claude-SearchBot are the ones that get you cited.
Does it block fake bots?
No. It only reports if a bot is verified, failed, unknown or unsupported. Blocking belongs in your WAF or CDN.
Why is my change not visible on the live site?
If Varnish, Fastly or another CDN caches robots.txt, purge it after you change the settings.
Does it work with Hyvä?
Yes. robots.txt is built on the server, so the theme does not matter.
Related
- AEO Audit — score 20 AI-visibility signals from the command line
- llms.txt — generate llms.txt, llms.jsonl, Markdown mirrors and agents.md
- AI Agent Activity — report on AI agent calls, crawler visits, fake bots and agent-built orders
- AI crawlers for Magento 2 (guide)
- Spoofed AI crawler — definition
- Web Bot Auth — definition
- All module reference pages · Compatibility matrix
This page was written from the README.md, composer.json and etc/*.xml files of tag 4.0.1, read on 16 October 2026. Settings, commands and defaults come straight from that code. If a newer release is out, the README on GitHub wins.