AI crawlers are the named user agents that AI companies send to read websites — a separate one for model training, for AI search indexing, and for fetching a page when a person asks about it. They are controlled independently in robots.txt, and blocking the training crawler does not remove a store from AI search answers.
Most Magento stores that are invisible in ChatGPT did not decide to be. They inherited a robots.txt written for Googlebot in 2015, and the agents that decide whether a store appears in an AI answer were never named in it. This page lists every agent that matters, what each one actually controls, and the four errors that account for nearly all accidental blocking.
OpenAI
OpenAI runs four agents. Settings are independent of one another: a site can allow search indexing while refusing training use. When both are permitted, OpenAI says one crawl may serve both purposes rather than fetching twice. After a robots.txt change, expect roughly 24 hours before search behaviour reflects it.
| User agent | What it controls | Effect of blocking | Published IPs |
|---|---|---|---|
| OAI-SearchBot | Whether the site can be surfaced in ChatGPT’s search features. This is the agent that governs ChatGPT visibility. | The site will not be shown in ChatGPT search answers, though it can still appear as a navigational link. | openai.com/searchbot.json |
| GPTBot | Whether content may be used to train OpenAI’s generative foundation models. | Content is excluded from future model training. No effect on ChatGPT search. | openai.com/gptbot.json |
| ChatGPT-User | Fetches a page when a person asks ChatGPT or a Custom GPT about it. Not used for automatic crawling. | Because the action is initiated by a person, robots.txt rules may not apply. Not used to determine Search appearance. | openai.com/chatgpt-user.json |
| OAI-AdsBot | Checks landing pages submitted as ads on ChatGPT against OpenAI’s policies. | Only visits pages submitted as ads. Data collected is not used for model training. | openai.com/adsbot.json |
Match on the agent token, not the whole user-agent string — OpenAI revises the surrounding boilerplate and bumps version numbers. One detail worth building into log analysis: when fetching robots.txt itself, OpenAI may add a robots.txt marker to the user-agent string, so that site owners whose logs omit paths can still tell those requests apart.
Anthropic
Anthropic runs three agents, documented separately since February 2026. The important divergence from OpenAI: Anthropic states that all three honour robots.txt, including the user-initiated fetcher.
| User agent | What it controls | Effect of blocking |
|---|---|---|
| ClaudeBot | Collection of public content that may be used to train Anthropic’s models. | Future content is excluded from training datasets. |
| Claude-SearchBot | Indexing that improves the quality and relevance of Claude’s search results. | Content is not indexed for search, which Anthropic says may reduce visibility and accuracy in Claude’s answers. |
| Claude-User | Retrieval when a person asks Claude something that requires reading a page. | Anthropic cannot fetch the page in response to user queries, which may reduce visibility in user-directed search. |
Two operational notes. Anthropic supports the non-standard Crawl-delay directive. And Anthropic does not publish IP ranges — its agents use public cloud provider addresses, so IP-based blocking is unreliable and can cut off access to robots.txt itself. Directives are needed for each agent and each subdomain separately.
Perplexity
| User agent | What it controls | Effect of blocking |
|---|---|---|
| PerplexityBot | Indexing pages so they can be cited in Perplexity answers. | Content is not indexed for citation. |
| Perplexity-User | Real-time retrieval triggered by a person’s question. | robots.txt generally does not apply to this agent. |
Perplexity’s crawler compliance has been publicly disputed. Cloudflare has documented cases of undeclared crawlers reaching sites that had blocked PerplexityBot. Treat robots.txt as a visibility control here, not an access control — that is true of every agent on this page, but it is least theoretical with this one.
Google is where the most confident wrong advice circulates, because three separate mechanisms get treated as one.
| Mechanism | What it actually governs |
|---|---|
| Googlebot | The Search index. AI Overviews and AI Mode are served from that index, so blocking Googlebot removes a store from Search and from Google’s AI answers. |
| Google-Extended | A product token, not a separate crawler — no page is ever fetched by it. It governs whether content Google already crawled may be used to train future Gemini models and for grounding in Gemini and Vertex AI products. Google states it does not affect inclusion in Google Search and is not a ranking signal. |
| Search Console “Search generative AI” control | A per-property toggle governing eligibility for AI Overviews, AI Mode, and AI Overviews in Discover. Launched UK-first on 3 June 2026 under a CMA mandate, effective 17 June. Availability was still limited at the time of writing — check Search Console rather than assuming access. |
Other agents worth naming
Lower priority for most Magento catalogues, but they belong in a complete file: Applebot-Extended (Apple Intelligence training opt-out), Amazonbot, Meta-ExternalAgent, Bytespider (ByteDance), and CCBot (Common Crawl — an open dataset that multiple AI systems train on, which makes it an indirect training route independent of every vendor agent above).
Four errors that cause almost all accidental blocking
1. Assuming a named agent inherits the wildcard rules
This is the one that silently breaks well-intentioned files, and it is a rule of the protocol, not a vendor quirk. Under RFC 9309, a crawler obeys only the single most specific matching group. A group naming GPTBot replaces the User-agent: * group entirely for that agent — it does not add to it.
So a file with a careful set of Disallow rules under *, followed by User-agent: GPTBot / Allow: /, tells GPTBot that the checkout, the account pages, the layered-navigation URLs and the internal search results are all fair game. The rules meant to protect them were never addressed to it.
Fix: repeat the full Disallow set inside the AI group, as in the file below.
2. Blocking GPTBot to stay out of ChatGPT
Wrong: GPTBot is ChatGPT’s crawler, so blocking it removes the store from ChatGPT.
Right: GPTBot governs training only. OAI-SearchBot is what determines whether a store can appear in ChatGPT search answers. They are different agents with different jobs, and the settings are independent.
The inverse error is more expensive: a store that allows GPTBot, blocks OAI-SearchBot, and concludes AEO does not work.
3. Expecting robots.txt to govern user-initiated fetches
ChatGPT-User and Perplexity-User fetch a page because a person asked for it. OpenAI states plainly that robots.txt rules may not apply to those requests, and Perplexity’s user agent generally does not honour them either. Anthropic is the exception — it says Claude-User does respect the file.
Practical consequence: a Disallow line for ChatGPT-User is not an access control. If content genuinely must not be readable, that is an authentication problem, not a robots.txt problem.
4. Believing Google-Extended controls AI Overviews
Wrong: disallowing Google-Extended keeps a store out of AI Overviews.
Right: AI Overviews are built from the Search index. Google-Extended governs training and Gemini grounding, and Google states it does not affect Search inclusion. Blocking it changes nothing about AI Overviews.
This error is repeated in a large share of published robots.txt guides, including recent ones. If a source tells you Google-Extended controls AI Overviews, treat the rest of that source with suspicion.
Where Magento 2 puts this
Magento generates robots.txt from design configuration, not from a file on disk. Edit it at Content → Design → Configuration, open the row for the relevant scope, and use Search Engine Robots → Edit custom instruction of robots.txt File.
Two things that catch people out. The file is served per domain, so a multi-website installation needs the instruction set on each website scope rather than once at default. And Magento’s own default is a wildcard group with restrictive Disallow rules — which, by the rule in error 1 above, is exactly the configuration that quietly blocks every AI agent that is not named.
The angeo/module-robots-txt-aeo module manages these groups with RFC 9309-safe parsing and validates the result from the CLI, if you would rather not hand-edit the field on every deployment.
A Magento robots.txt that behaves correctly
This allows AI search and training access to public catalogue content while keeping every agent out of the paths that waste crawl budget or expose customer surfaces. Note that the disallow set is repeated inside the AI group — that repetition is the point, not redundancy.
# --- Conventional crawlers -----------------------------------
User-agent: *
Allow: /
Disallow: /checkout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /catalogsearch/
Disallow: /sales/
Disallow: /wishlist/
Disallow: /review/
Disallow: /*?SID=
Disallow: /*?limit=
Disallow: /*?dir=
Disallow: /*?order=
Disallow: /*?p=
# --- AI agents ------------------------------------------------
# One group, several agents, identical rules.
# Named groups do NOT inherit from "*" (RFC 9309), so the
# disallow set above is repeated here deliberately.
User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: CCBot
Allow: /
Disallow: /checkout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /catalogsearch/
Disallow: /sales/
Disallow: /wishlist/
Disallow: /review/
Disallow: /*?SID=
Disallow: /*?limit=
Disallow: /*?dir=
Disallow: /*?order=
Disallow: /*?p=
Sitemap: https://example.com/sitemap.xml
Replace the sitemap URL, and drop any agent whose access you have deliberately decided against — the file is a policy statement, and the right answer differs between a brand that wants maximum AI reach and a publisher protecting licensed content.
Verifying that it worked
A file that reads correctly and a crawler that is actually being served are different claims. Three checks, in order of how often they catch something:
Server logs, filtered by agent token. The most common failure is not the file at all — it is a WAF, CDN bot rule, or rate limiter returning 403 or 429 to a crawler that robots.txt permits. A store can have a perfect file and still be unreachable. Look for the token, then look at the status codes it received.
IP verification for OpenAI agents. The user-agent string is trivially spoofed. OpenAI publishes JSON ranges per agent at the URLs in the table above; match request IPs against those before trusting the label. Anthropic publishes no ranges, so token matching is all that is available there.
Referral parameters. ChatGPT referrals carry utm_source=chatgpt.com, which gives a measurable signal in analytics once OAI-SearchBot has been allowed long enough to matter.
Questions
Does blocking GPTBot remove my store from ChatGPT?
No. GPTBot controls training only. Whether a store can appear in ChatGPT search answers is governed by OAI-SearchBot, and the two settings are independent of each other.
Why do AI crawlers ignore the Disallow rules I already have?
They do not ignore them — they never received them. Under RFC 9309 a crawler obeys only the most specific group that matches its token, so naming an agent in its own group replaces the wildcard group for that agent instead of adding to it. Repeat the disallow set inside the AI group.
Can I stop ChatGPT from reading a page a user pastes into it?
Not with robots.txt. ChatGPT-User fetches on a person’s instruction, and OpenAI states that robots.txt rules may not apply to user-initiated actions. Content that must not be readable needs authentication, not a crawl directive.
Does disallowing Google-Extended remove my store from AI Overviews?
No. AI Overviews are built from Google’s Search index, which is populated by Googlebot. Google-Extended governs training and Gemini grounding, and Google states it does not affect inclusion in Search.
How long after changing robots.txt does anything happen?
OpenAI says roughly 24 hours for search behaviour to reflect a change. Other vendors do not publish a figure. Recrawl of the catalogue itself takes longer and depends on the site’s crawl rate.
Should a Magento store allow training crawlers as well as search crawlers?
For most merchants, yes. A model that has seen the brand describes it more accurately even when it is not browsing, and blocking training gains a store nothing it can measure. The case for refusing is real but narrow: licensed or paywalled content, or a contractual reason to withhold it.
Sources
- OpenAI — Overview of OpenAI Crawlers,
developers.openai.com/api/docs/bots. Agent roles, user-agent strings, IP range files, the 24-hour figure, and the statement on user-initiated actions. - Anthropic — support article on web crawling and blocking,
support.anthropic.com. The three agents, their effects when disabled, Crawl-delay support, and the position on IP blocking. - RFC 9309, Robots Exclusion Protocol — group matching and specificity.
- Google — crawler and product token documentation, on Google-Extended’s scope and its independence from Search.
- Reporting on Perplexity crawler behaviour, including Cloudflare’s documentation of undeclared crawlers.