Claude Crawler Update
Anthropic now gives website owners more granular control over training, AI search indexing, and user-initiated Claude access.
The short version
Anthropic now separates its website crawlers by purpose. This means you can make a more precise decision: block content from training-related crawling while keeping public pages available for Claude's AI search and user-requested page access.
What changed?
Anthropic's crawler documentation now separates Claude-related access into distinct bots. Previously, many site owners treated ClaudeBot as a single all-purpose crawler. The newer model makes it possible to control training, search indexing, and user-triggered retrieval independently.
Why this matters for GEO
A broad rule that blocks every Claude crawler can affect more than training access. It may also limit whether public content can be indexed for Claude search experiences or retrieved when a user asks Claude to visit a specific page.
The three Claude crawlers
ClaudeBot
Training-relatedClaudeBot collects public web content that may be used to train and improve Anthropic's AI models. This is the crawler to control if your concern is training-related access.
User-agent: ClaudeBot
Claude-SearchBot
AI search indexingClaude-SearchBot is intended to index public website content for Claude's search experiences. If you want public pages to remain eligible for discovery in Claude search, check that this user agent is not unintentionally blocked.
User-agent: Claude-SearchBot
Claude-User
User-requested fetchingClaude-User may request a page when a Claude user asks the product to access a specific URL or retrieve current information. This is different from ongoing search indexing.
User-agent: Claude-User
Example robots.txt policy
Your crawler policy should reflect your own business, legal, privacy, and publishing requirements. The example below blocks training-related crawling while keeping public content available for Claude search indexing and user-requested access.
# Block training-related crawling
User-agent: ClaudeBot
Disallow: /
# Allow indexing for Claude search
User-agent: Claude-SearchBot
Allow: /
# Allow user-requested access to public pages
User-agent: Claude-User
Allow: /
# Keep sensitive areas restricted
User-agent: *
Disallow: /admin/
Disallow: /dashboard/
Disallow: /account/Important
This is an example, not universal advice. Do not use robots.txt as your only security control. Private or sensitive content should require authentication and should not be publicly accessible in the first place.
What website owners should do now
Audit existing Claude rules
Check whether your robots.txt contains a broad ClaudeBot, anthropic-ai, or wildcard block that also affects your intended search visibility.
Separate training from search decisions
Decide independently whether to allow ClaudeBot, Claude-SearchBot, and Claude-User based on their different roles.
Keep public content genuinely public
Key pages should not require login, return server errors, or depend on empty JavaScript shells to show their main content.
Use clear content structure
Give important pages direct answers, descriptive headings, accurate information, and clear organization or author context.
Monitor changes over time
Use a repeatable set of relevant prompts to observe mentions, citations, and competitor sources in Claude and other AI search systems.
Visibility is not guaranteed
Allowing Claude-SearchBot removes one potential access barrier. It does not guarantee crawling, indexing, retrieval, citation, or recommendation. Source selection can change based on the question, competing pages, freshness, content quality, and Claude's own search systems.
Free GazeRank tool
Check your AI crawler access
Use the GazeRank robots.txt Checker to see whether AI crawlers, including Claude-related user agents, appear allowed or blocked on your website.
Related learning