AI Crawlers Compared: GPTBot, ClaudeBot, PerplexityBot, and Google-Extended
Each major AI company now runs at least two or three separate crawlers for different jobs — training, search indexing, and live user requests. Blocking the wrong one has a very different effect than blocking the right one. Here is what each one actually does.
"AI bot" is not one thing
A single Disallow: / rule aimed at a company's main crawler often only blocks training data collection. It does not automatically block that same company's search or citation crawler — those are separate user-agents with separate rules. Getting this wrong is the single most common cause of a site accidentally disappearing from AI-generated answers while its owner believed AI access was already blocked (or vice versa).
The major AI crawlers, side by side
Four companies, three jobs each (mostly). This table shows the current publicly documented crawler for each purpose — check each company's own documentation before deploying a strict policy, since crawler names and behavior continue to change.
| Company | Training crawler | Search / citation crawler | User-triggered fetcher |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | — (no declared training crawler) | PerplexityBot | Perplexity-User |
| Google-Extended | Googlebot (shared with standard Search) | — (AI Overviews use the standard Googlebot index) |
Does each one actually respect robots.txt?
OpenAI
GPTBot and OAI-SearchBot honor robots.txt. OpenAI's own documentation notes ChatGPT-User may not be governed by robots.txt the same way, since a person — not an automated schedule — triggered the fetch.
Anthropic
Anthropic states all three of its bots honor robots.txt, including the user-triggered one — the one exception among the four companies here.
Perplexity
PerplexityBot is documented to respect robots.txt. Perplexity-User is treated as user-directed and may not. Cloudflare has separately reported undeclared Perplexity crawler activity that ignored disallow rules entirely — see Further Reading.
Google-Extended honors robots.txt and can be disallowed independently to opt out of Gemini training, without affecting your regular Google Search ranking.
Compliance with robots.txt is a voluntary norm, not a technical enforcement mechanism — a directive only works if the crawler chooses to read and honor it, and a spoofed user-agent can claim to be anything. Treat robots.txt as the correct first step for any crawler that documents support for it, and treat a WAF or server-level rule as the real backstop for anything that doesn't.
A reasonable starting configuration
There's no single right answer — allowing training crawlers means your content may eventually inform a model's general knowledge of your brand; blocking them keeps your content out of training data while usually still allowing search/citation crawlers to surface and cite you. A common middle-ground policy:
# Keep AI search & citation crawlers allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
# Decide separately whether to allow training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Keep sensitive paths blocked for every bot, including humans
User-agent: *
Disallow: /admin/
Disallow: /account/
Disallow: /checkout/Allow changes can take roughly a day to be reflected in a company's live crawling and search behavior, so don't judge a robots.txt change by checking again five minutes later.
Common mistakes
- Leaving a blanket "Disallow: /" under a training-bot token from an old 2023-era robots.txt, which also blocks that company's search/citation bot if you never split the rules apart.
- Assuming one company's crawler behavior applies to all of them — Anthropic's user-triggered fetcher respects robots.txt; OpenAI's and Perplexity's may not.
- Blocking a bot in robots.txt while a CDN or WAF (Cloudflare, Sucuri, etc.) is separately configured to challenge or block the same user-agent — the robots.txt rule then has no effect either way.
- Treating IP-based blocking as a reliable substitute for robots.txt — several providers explicitly warn that their crawlers run on shared cloud IP ranges, so blocking by IP can be unreliable and may even prevent the crawler from reading your robots.txt file in the first place.
Next step
Check which AI crawlers can actually reach your site
Run the AI Crawler Checker and robots.txt Checker to see exactly which of these user-agents are allowed, blocked, or silently caught by a CDN default you didn't set.
Check my site →