GPTBot vs OAI-SearchBot – Definition and Difference
GPTBot and OAI-SearchBot are two separate OpenAI crawlers with different jobs: GPTBot collects data that may be used to train models, while OAI-SearchBot builds the index that ChatGPT search draws on when it answers a question. Blocking GPTBot does not remove a site from ChatGPT search results. Blocking OAI-SearchBot does. Most stores that set out to “block AI” block the first and keep the second – the opposite of what they intended, in both directions.
GPTBot is about training; OAI-SearchBot is about being findable. They are configured independently in robots.txt, and confusing them is the most common AI-crawler mistake in ecommerce.
The OpenAI agents a store meets
| User-agent | Purpose | Blocking it means |
|---|---|---|
GPTBot | Crawls content that may be used for model training | Your content is excluded from training. Search visibility unchanged |
OAI-SearchBot | Builds the ChatGPT search index | You disappear from ChatGPT search results |
ChatGPT-User | Fetches a page live because a user asked about it in a conversation | Little. OpenAI says robots.txt rules may not apply to these user-initiated fetches |
OpenAI also runs OAI-AdsBot, which only checks landing pages submitted as ChatGPT ads. The full list, with published IP ranges, is in AI crawlers for Magento 2.
The settings are independent. A store can allow search while declining training, which is what most merchants actually want:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /If you want to say what AI systems may do with content after fetching it – quote it, but not train on it – use Content Signals alongside these rules.
What we measured
Across 762 reachable Magento 2 storefronts from the Tranco top-200,000 frame (list ID ZJGPG, scan of 9 August 2026):
| Crawler blocked | Stores | Share |
|---|---|---|
| At least one AI crawler | 110 | 14% |
| GPTBot | ~99 | 13% |
| ClaudeBot | ~99 | 13% |
| PerplexityBot | ~23 | 3% |
Method: a store counts as blocking a crawler when its robots.txt contains a group that applies to that user-agent with a Disallow: / covering the root. 23 stores hit rate limits during the run and are excluded from the numerator, so these are floors.
The RFC 9309 trap
Under RFC 9309, a crawler obeys the most specific single group that names it – it does not combine rules across groups. A file with User-agent: GPTBot followed by a separate User-agent: * group containing the store’s real Disallow list gives GPTBot only the rules in its own group. Every restriction in the wildcard group is silently dropped for that bot.
The fix is repetition, not inheritance: each AI-bot group must carry the full Disallow set it needs.
This is why a robots.txt can look like it blocks a crawler and not block it. It is also why hand-editing the file at scale is a poor idea.
robots.txt is a request, not a control
Compliance is voluntary. robots.txt keeps well-behaved crawlers out; it does not stop anything that ignores it. Two further limits worth knowing:
- User-agent strings are trivially spoofed. Verifying a crawler means checking the request’s IP against the vendor’s published ranges, or a request signature where the vendor offers one (see Web Bot Auth).
- Disallow does not remove existing knowledge. Content already trained on, or already indexed, is unaffected by a rule added today.
Magento’s default is a problem
Stock Magento 2 ships no AI-crawler rules, and many hosts and extensions add broad Disallow patterns that catch AI agents as collateral. The result is a store that never decided anything about AI access and is excluded anyway.
angeo/module-robots-txt-aeo manages AI-crawler rules with lossless RFC 9309 round-trip parsing – it does not rewrite the rest of your file – and ships IP verification:
bin/magento angeo:robots:verify-bot-ip 12.34.56.78Questions
- What is the difference between GPTBot and OAI-SearchBot?
- GPTBot crawls content that may be used for model training. OAI-SearchBot builds the index behind ChatGPT search. They are separate crawlers configured independently, and blocking one has no effect on the other.
- Does blocking GPTBot remove my store from ChatGPT?
- No. GPTBot governs training data. Removing a store from ChatGPT search results requires blocking OAI-SearchBot, a different user-agent with a different job.
- Which OpenAI crawlers should an ecommerce store allow?
- Most merchants want OAI-SearchBot allowed so the store can appear in ChatGPT search. Whether to allow GPTBot is a separate decision about training data. ChatGPT-User fetches pages when a user asks, and OpenAI says robots.txt may not apply to it.
- Why does my robots.txt block a bot that still crawls me?
- Usually because RFC 9309 groups do not merge. A crawler obeys only the single most specific group naming it, so restrictions in a wildcard group do not apply to a bot that has its own group. Each group must repeat the full Disallow set.
- Can I trust the user-agent string a crawler sends?
- No, it is trivially spoofed. Verification means checking the request IP against the vendor’s published ranges.