Spoofed AI Crawler – Definition and How to Tell a Real Bot
A spoofed AI crawler is a request that says “I am GPTBot” (or ClaudeBot, or PerplexityBot) in its User-Agent header, but does not come from that vendor. The header is one line of text that anyone can send.
The User-Agent is a claim; the vendor’s published IP range or a valid signature is the proof.
Why it matters for a store
Spoofed bots cause three problems:
- Wrong numbers. If you count every “GPTBot” request, your AI traffic report includes scrapers that only borrowed the name.
- Wrong decisions. A WAF rule that trusts the User-Agent lets fake bots through, while a rule that blocks by name can also block the real one.
- Wasted load. Aggressive scrapers often hide behind well-known bot names because many sites allow them.
How to check a bot
There are two kinds of proof. A store can use whichever the vendor publishes.
| Check | How it works | Strength |
|---|---|---|
| IP range | The vendor publishes the addresses its crawler uses. The request’s source IP must be in that list. | Good, if your server sees the real client IP |
| Signed request (Web Bot Auth) | The bot signs each request (RFC 9421). The site checks the signature against the vendor’s published key. | Strongest, but few vendors sign today |
| User-Agent only | Read the header | No proof at all |
What vendors publish (October 2026)
| Vendor | IP ranges | Signed requests |
|---|---|---|
| OpenAI | Yes | Yes (ChatGPT agent) |
| Anthropic | Yes — one shared list for ClaudeBot, Claude-User and Claude-SearchBot | No |
| Perplexity | Yes, per bot | No |
| Yes, for user-triggered fetchers such as Google-Agent. Google-Extended is a token, not a crawler. | No | |
| Apple, Meta, ByteDance, Mistral, DuckDuckGo | No AI crawler list | No |
A match in a shared list proves the vendor, not which of its bots sent the request. Say that plainly in any report.
Four results, not two
An honest check has four answers:
- verified — the IP is in the vendor’s range, or the signature is valid;
- failed / spoofed — it is not;
- unknown — no decision: the list could not be loaded, or the server only sees a private proxy address;
- unsupported — the vendor publishes no way to check.
“Unknown” is not “spoofed”. Behind a CDN or load balancer, set up trusted proxy headers first, or every check ends as unknown.
A Magento example
AI Agent Activity downloads the IP lists of OpenAI, Anthropic, Perplexity and Google-Agent once a day and marks each AI crawler request as verified or spoofed. It uses the IP for the check and never stores it. Robots.txt AEO adds two CLI checks: angeo:robots:verify-bot-ip for an address and angeo:robots:verify-bot-request for a signed request. Both modules report. Blocking belongs in your WAF or CDN.
Questions
- Can I trust the User-Agent header?
- No. It is a claim that any client can send. Use the vendor’s published IP ranges or a request signature as proof.
- Should I block spoofed AI crawlers?
- Usually yes, at the WAF or CDN, once you are sure the check sees the real client IP. Do not block real search bots by mistake: they bring AI citations.
- Why does my check say unknown so often?
- Most often the server only sees a proxy or load balancer address. Configure trusted proxy headers so the application sees the client IP.
- Does blocking the IPs of a real AI crawler opt me out?
- It stops the crawler, but it also stops it reading robots.txt, which is where your preference lives. Use robots.txt rules to opt out.
Related
- GPTBot vs OAI-SearchBot
- Web Bot Auth – definition
- AI crawlers reference
- Robots.txt AEO module
- AI Agent Activity module
- Glossary