Last verified: 23 September 2026 against RFC 9309 and the crawler documentation of OpenAI, Anthropic, Perplexity and Google.
Short answer: allow the AI search crawlers and the user-triggered fetchers, keep Googlebot allowed, and decide about training crawlers separately. Give each AI group the same Disallow lines as your User-agent: * group, because a named group does not inherit them. Where the groups sit in the file does not matter.
Your pages are fast and your catalog is clean, but ChatGPT still does not name your store. Often the reason is robots.txt: a file written for Google years ago that never named the crawlers AI assistants use today.
Why robots.txt comes first
robots.txt is a gate. Structured data, llms.txt and a product feed only help once a crawler is allowed in. OpenAI states it plainly in its crawler documentation: sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers.
Three kinds of AI crawler
| Kind | What it does | Examples | For a store |
|---|---|---|---|
| Search | Builds the index an assistant searches when it answers. Can cite and link your pages. | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Allow |
| User-triggered | Opens a page because a person asked the assistant about it. | ChatGPT-User, Claude-User, Perplexity-User | Allow |
| Training | Collects content to train future models. No link back. | GPTBot, ClaudeBot, CCBot | Your choice |
Google is different. Google has no separate AI crawler. AI Overviews and AI Mode are built from the normal Googlebot index. Google-Extended is a robots.txt token, not a crawler: it only controls whether Gemini may use content for training and grounding. Blocking it does not remove you from AI Overviews. To limit AI features in Search, Google points to nosnippet, max-snippet and the Search generative AI setting in Search Console (Google: AI features and your website).
The most common mistake: blocking GPTBot to “leave ChatGPT”. It does not work, because ChatGPT search uses OAI-SearchBot. More on this in GPTBot vs OAI-SearchBot.
The crawlers that matter in 2026
| User-agent | Company | Kind | If you block it |
|---|---|---|---|
OAI-SearchBot | OpenAI | Search | Not shown in ChatGPT search answers. |
ChatGPT-User | OpenAI | User-triggered | ChatGPT may not open your page for a user. OpenAI says robots.txt may not apply to these fetches. |
GPTBot | OpenAI | Training | Content left out of future training. No effect on ChatGPT search. |
Claude-SearchBot | Anthropic | Search | Not indexed for Claude’s search answers. |
Claude-User | Anthropic | User-triggered | Claude cannot open your page for a user. Anthropic says it respects robots.txt. |
ClaudeBot | Anthropic | Training | Content left out of future training. |
PerplexityBot | Perplexity | Search | Not indexed for Perplexity answers. |
Perplexity-User | Perplexity | User-triggered | Robots.txt generally does not apply, per Perplexity. |
Googlebot | Search (incl. AI Overviews, AI Mode) | Gone from Google Search and its AI features. | |
Google-Extended | Token, not a crawler | No Gemini training or grounding. Search and AI Overviews unchanged. | |
Applebot-Extended | Apple | Token, not a crawler | No training of Apple’s models. Crawling is done by Applebot. |
Claude-Web and anthropic-ai are old Anthropic names. You can leave them in the file, but they are not the crawlers that matter now. The full, dated list is in AI crawlers for Magento 2.
The rule most robots.txt guides get wrong
Under RFC 9309, a crawler follows only the one most specific group that names it. Groups do not merge, and their order in the file does not matter.
# WRONG - looks safe, is not User-agent: * Disallow: /checkout/ Disallow: /customer/ User-agent: OAI-SearchBot Allow: / # OAI-SearchBot reads ONLY its own group. # /checkout/ and /customer/ are now open to it. # RIGHT - the named group repeats the Disallow lines User-agent: OAI-SearchBot Allow: / Disallow: /checkout/ Disallow: /customer/
Inside one group the longest matching path wins, so Disallow: /checkout/ still beats Allow: /.
A safe robots.txt for Magento 2
This file keeps private and duplicate URLs closed for everyone, opens the catalog to AI search, and puts training in its own group so you can change it with one edit. Several user-agents can share one group.
# --- Everyone without a named group (Googlebot, Bingbot, ...) --- User-agent: * Allow: / Disallow: /checkout/ Disallow: /customer/ Disallow: /catalogsearch/ Disallow: /sales/ Disallow: /wishlist/ Disallow: /review/ Disallow: /*?SID= Disallow: /*?___store= Disallow: /*?limit= Disallow: /*?dir= Disallow: /*?order= # --- AI search crawlers and user-triggered fetchers --- # Named groups do not inherit from "*": the Disallow set is repeated. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / Disallow: /checkout/ Disallow: /customer/ Disallow: /catalogsearch/ Disallow: /sales/ Disallow: /wishlist/ Disallow: /review/ Disallow: /*?SID= Disallow: /*?___store= Disallow: /*?limit= Disallow: /*?dir= Disallow: /*?order= # --- Training: your choice. To opt out, replace the rules below with "Disallow: /" --- User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Google-Extended User-agent: Applebot-Extended Allow: / Disallow: /checkout/ Disallow: /customer/ Disallow: /catalogsearch/ Disallow: /sales/ Disallow: /wishlist/ Disallow: /review/ Disallow: /*?SID= Disallow: /*?___store= Disallow: /*?limit= Disallow: /*?dir= Disallow: /*?order= Sitemap: https://yourstore.com/sitemap.xml
Sitemap lines are for sitemaps only. Do not add Sitemap: .../llms.txt. Crawlers that support llms.txt look for it at the site root.
robots.txt says who may fetch a page. To say what AI may do with it – quote it, but not train on it – add a Content-Signal line.
Where Magento serves robots.txt from
Magento admin (most stores)
Magento builds robots.txt from design configuration. Go to Content → Design → Configuration, open the row for the website or store view that serves the domain, then Search Engine Robots → Edit custom instruction of robots.txt File. On a multi-website install, set it on each website: every domain serves its own file.
A physical pub/robots.txt
If admin changes do not show up, a static file is probably served first.
# Is a static file in the way?
ls -la pub/robots.txt
curl -sI https://yourstore.com/robots.txt
Edit that file, or remove it so the admin configuration is used.
Caches
Varnish, Cloudflare and Fastly can keep serving an old file. On Adobe Commerce Cloud robots.txt is served through Fastly: purge it after every change and check the live file, not the admin.
How to check that it works
curl -s https://yourstore.com/robots.txt – check each AI group has Allow: / and the full Disallow set.grep -E "OAI-SearchBot|Claude-SearchBot|PerplexityBot" access.log – then check the status codes. A WAF or CDN rule that returns 403 or 429 blocks a bot that robots.txt allows.bin/magento angeo:robots:verify-bot-ip <ip>.bin/magento angeo:aeo:audit checks the rules per store view, or use the free web scan to see the file from outside.To manage the groups from the admin without hand-editing, angeo/module-robots-txt-aeo parses robots.txt to RFC 9309, keeps your own rules byte for byte, and writes only its own block.
Questions
Which AI crawlers should a Magento store allow?
Allow the search crawlers and the user-triggered fetchers: OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User (Perplexity). Keep Googlebot allowed, because Google’s AI Overviews and AI Mode use the normal Search index. Training crawlers such as GPTBot and ClaudeBot are a separate choice.
Does blocking GPTBot remove my store from ChatGPT?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot. Blocking GPTBot only keeps your content out of future training.
Does Google-Extended control AI Overviews?
No. Google-Extended is a robots.txt token, not a crawler. It controls whether Google may use your content for Gemini training and grounding. AI Overviews and AI Mode use Googlebot. To limit them, use nosnippet, max-snippet or the Search generative AI setting in Search Console.
Does the order of groups in robots.txt matter?
No. Under RFC 9309 a crawler follows the one most specific group that names it, wherever it is in the file. A named group replaces the User-agent: * group for that crawler, so it must repeat the Disallow lines it needs.
Where do I edit robots.txt in Magento 2?
In the admin: Content, Design, Configuration. Open the row for the website or store view that serves the domain, then Search Engine Robots and Edit custom instruction of robots.txt File. If a physical pub/robots.txt exists, the web server serves that file instead.
How do I check that a crawler can really reach my store?
Check three things: the live file at /robots.txt, your server logs for the bot’s user-agent and the status codes it got, and the bot’s IP address against the ranges the vendor publishes. A WAF or CDN rule that returns 403 blocks a bot even when robots.txt allows it.
After robots.txt
Access is step one. What an assistant can do with your pages next depends on:
- Product JSON-LD with
offers.availability, rendered on the server – see Product schema for AI search. - llms.txt – a short map of your key pages. See what llms.txt is and what reads it.
- A product feed for ChatGPT shopping results – see ACP product feed.
The full order of work is in the Magento 2 AEO guide. Related: AI crawlers and Magento: common questions · Web Bot Auth.
See which AI crawlers your Magento store lets in – and every other AEO signal – in one scan.