# How to Fix robots.txt for ChatGPT and Gemini in Magento 2 > Which AI crawlers a Magento 2 store should allow, why named robots.txt groups must repeat Disallow rules (RFC 9309), and a safe file to copy. Source: https://angeo.dev/magento-2-robots-txt-chatgpt-gemini-ai-bots/ Updated: 2026-09-26 *Last verified: 23 September 2026 against RFC 9309 and the crawler documentation of OpenAI, Anthropic, Perplexity and Google.* **Short answer:** allow the AI *search* crawlers and the user-triggered fetchers, keep Googlebot allowed, and decide about *training* crawlers separately. Give each AI group the same `Disallow` lines as your `User-agent: *` group, because a named group does not inherit them. Where the groups sit in the file does not matter. Your pages are fast and your catalog is clean, but ChatGPT still does not name your store. Often the reason is `robots.txt`: a file written for Google years ago that never named the crawlers AI assistants use today. [image: How to fix robots.txt for ChatGPT and Gemini in Magento 2] ## Why robots.txt comes first robots.txt is a gate. Structured data, [llms.txt](https://angeo.dev/llms-txt-definition/) and a [product feed](https://angeo.dev/acp-product-feed-definition/) only help once a crawler is allowed in. OpenAI states it plainly in its [crawler documentation](https://developers.openai.com/api/docs/bots): sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers. ## Three kinds of AI crawler | Kind | What it does | Examples | For a store | | **Search** | Builds the index an assistant searches when it answers. Can cite and link your pages. | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot` | Allow | | **User-triggered** | Opens a page because a person asked the assistant about it. | `ChatGPT-User`, `Claude-User`, `Perplexity-User` | Allow | | **Training** | Collects content to train future models. No link back. | `GPTBot`, `ClaudeBot`, `CCBot` | Your choice | **Google is different.** Google has no separate AI crawler. AI Overviews and AI Mode are built from the normal `Googlebot` index. `Google-Extended` is a robots.txt token, not a crawler: it only controls whether Gemini may use content for training and grounding. Blocking it does not remove you from AI Overviews. To limit AI features in Search, Google points to `nosnippet`, `max-snippet` and the Search generative AI setting in Search Console ([Google: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)). The most common mistake: blocking `GPTBot` to "leave ChatGPT". It does not work, because ChatGPT search uses `OAI-SearchBot`. More on this in [GPTBot vs OAI-SearchBot](https://angeo.dev/gptbot-vs-oai-searchbot/). ## The crawlers that matter in 2026 | User-agent | Company | Kind | If you block it | | `OAI-SearchBot` | OpenAI | Search | Not shown in ChatGPT search answers. | | `ChatGPT-User` | OpenAI | User-triggered | ChatGPT may not open your page for a user. OpenAI says robots.txt may not apply to these fetches. | | `GPTBot` | OpenAI | Training | Content left out of future training. No effect on ChatGPT search. | | `Claude-SearchBot` | Anthropic | Search | Not indexed for Claude's search answers. | | `Claude-User` | Anthropic | User-triggered | Claude cannot open your page for a user. Anthropic says it respects robots.txt. | | `ClaudeBot` | Anthropic | Training | Content left out of future training. | | `PerplexityBot` | Perplexity | Search | Not indexed for Perplexity answers. | | `Perplexity-User` | Perplexity | User-triggered | Robots.txt generally does not apply, per Perplexity. | | `Googlebot` | Google | Search (incl. AI Overviews, AI Mode) | Gone from Google Search *and* its AI features. | | `Google-Extended` | Google | Token, not a crawler | No Gemini training or grounding. Search and AI Overviews unchanged. | | `Applebot-Extended` | Apple | Token, not a crawler | No training of Apple's models. Crawling is done by `Applebot`. | `Claude-Web` and `anthropic-ai` are old Anthropic names. You can leave them in the file, but they are not the crawlers that matter now. The full, dated list is in [AI crawlers for Magento 2](https://angeo.dev/docs/ai-crawlers/). ## The rule most robots.txt guides get wrong Under [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), a crawler follows **only the one most specific group** that names it. Groups do not merge, and their order in the file does not matter. ``` # WRONG - looks safe, is not User-agent: * Disallow: /checkout/ Disallow: /customer/ User-agent: OAI-SearchBot Allow: / # OAI-SearchBot reads ONLY its own group. # /checkout/ and /customer/ are now open to it. # RIGHT - the named group repeats the Disallow lines User-agent: OAI-SearchBot Allow: / Disallow: /checkout/ Disallow: /customer/ ``` Inside one group the longest matching path wins, so `Disallow: /checkout/` still beats `Allow: /`. ## A safe robots.txt for Magento 2 This file keeps private and duplicate URLs closed for everyone, opens the catalog to AI search, and puts training in its own group so you can change it with one edit. Several user-agents can share one group. ``` # --- Everyone without a named group (Googlebot, Bingbot, ...) --- User-agent: * Allow: / Disallow: /checkout/ Disallow: /customer/ Disallow: /catalogsearch/ Disallow: /sales/ Disallow: /wishlist/ Disallow: /review/ Disallow: /*?SID= Disallow: /*?___store= Disallow: /*?limit= Disallow: /*?dir= Disallow: /*?order= # --- AI search crawlers and user-triggered fetchers --- # Named groups do not inherit from "*": the Disallow set is repeated. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / Disallow: /checkout/ Disallow: /customer/ Disallow: /catalogsearch/ Disallow: /sales/ Disallow: /wishlist/ Disallow: /review/ Disallow: /*?SID= Disallow: /*?___store= Disallow: /*?limit= Disallow: /*?dir= Disallow: /*?order= # --- Training: your choice. To opt out, replace the rules below with "Disallow: /" --- User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Google-Extended User-agent: Applebot-Extended Allow: / Disallow: /checkout/ Disallow: /customer/ Disallow: /catalogsearch/ Disallow: /sales/ Disallow: /wishlist/ Disallow: /review/ Disallow: /*?SID= Disallow: /*?___store= Disallow: /*?limit= Disallow: /*?dir= Disallow: /*?order= Sitemap: https://yourstore.com/sitemap.xml ``` **Sitemap lines are for sitemaps only.** Do not add `Sitemap: .../llms.txt`. Crawlers that support llms.txt look for it at the site root. robots.txt says who may *fetch* a page. To say what AI may *do* with it - quote it, but not train on it - add a [Content-Signal line](https://angeo.dev/content-signals-definition/). ## Where Magento serves robots.txt from ### Magento admin (most stores) Magento builds `robots.txt` from design configuration. Go to **Content -> Design -> Configuration**, open the row for the website or store view that serves the domain, then **Search Engine Robots -> Edit custom instruction of robots.txt File**. On a multi-website install, set it on each website: every domain serves its own file. ### A physical pub/robots.txt If admin changes do not show up, a static file is probably served first. ``` # Is a static file in the way? ls -la pub/robots.txt curl -sI https://yourstore.com/robots.txt ``` Edit that file, or remove it so the admin configuration is used. ### Caches Varnish, Cloudflare and Fastly can keep serving an old file. On Adobe Commerce Cloud robots.txt is served through Fastly: purge it after every change and check the live file, not the admin. ## How to check that it works 01 Read the live file `curl -s https://yourstore.com/robots.txt` - check each AI group has `Allow: /` and the full Disallow set. 02 Look at the logs `grep -E "OAI-SearchBot|Claude-SearchBot|PerplexityBot" access.log` - then check the status codes. A WAF or CDN rule that returns 403 or 429 blocks a bot that robots.txt allows. 03 Verify the IP A user-agent string is easy to fake. Check request IPs against the ranges OpenAI and other vendors publish (Anthropic publishes none), or run `bin/magento angeo:robots:verify-bot-ip `. 04 Run the audit `bin/magento angeo:aeo:audit` checks the rules per store view, or use the [free web scan](https://angeo.dev/ai-magento-audit/) to see the file from outside. To manage the groups from the admin without hand-editing, [angeo/module-robots-txt-aeo](https://angeo.dev/modules/robots-txt-aeo/) parses robots.txt to RFC 9309, keeps your own rules byte for byte, and writes only its own block. ## Questions ### Which AI crawlers should a Magento store allow? Allow the search crawlers and the user-triggered fetchers: OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User (Perplexity). Keep Googlebot allowed, because Google's AI Overviews and AI Mode use the normal Search index. Training crawlers such as GPTBot and ClaudeBot are a separate choice. ### Does blocking GPTBot remove my store from ChatGPT? No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot. Blocking GPTBot only keeps your content out of future training. ### Does Google-Extended control AI Overviews? No. Google-Extended is a robots.txt token, not a crawler. It controls whether Google may use your content for Gemini training and grounding. AI Overviews and AI Mode use Googlebot. To limit them, use nosnippet, max-snippet or the Search generative AI setting in Search Console. ### Does the order of groups in robots.txt matter? No. Under RFC 9309 a crawler follows the one most specific group that names it, wherever it is in the file. A named group replaces the User-agent: * group for that crawler, so it must repeat the Disallow lines it needs. ### Where do I edit robots.txt in Magento 2? In the admin: Content, Design, Configuration. Open the row for the website or store view that serves the domain, then Search Engine Robots and Edit custom instruction of robots.txt File. If a physical pub/robots.txt exists, the web server serves that file instead. ### How do I check that a crawler can really reach my store? Check three things: the live file at /robots.txt, your server logs for the bot's user-agent and the status codes it got, and the bot's IP address against the ranges the vendor publishes. A WAF or CDN rule that returns 403 blocks a bot even when robots.txt allows it. ## After robots.txt Access is step one. What an assistant can do with your pages next depends on: - **Product JSON-LD** with [`offers.availability`](https://angeo.dev/offers-availability-definition/), rendered on the server - see [Product schema for AI search](https://angeo.dev/magento-2-product-schema-json-ld-ai-search/). - **llms.txt** - a short map of your key pages. See [what llms.txt is and what reads it](https://angeo.dev/llms-txt-definition/). - **A product feed** for ChatGPT shopping results - see [ACP product feed](https://angeo.dev/acp-product-feed-definition/). The full order of work is in the [Magento 2 AEO guide](https://angeo.dev/magento-2-aeo-guide/). Related: [AI crawlers and Magento: common questions](https://angeo.dev/ai-crawlers-magento-questions/) - [Web Bot Auth](https://angeo.dev/web-bot-auth-definition/). See which AI crawlers your Magento store lets in - and every other AEO signal - in one scan. [Run the free scan ->](https://angeo.dev/ai-magento-audit/) [Install the CLI audit](https://packagist.org/packages/angeo/module-aeo-audit) ## More on this topic - [How to Control AI Overviews and AI Mode on Magento](https://angeo.dev/control-ai-overviews-magento/)