PerplexityBot Crawl Update
What Perplexity’s robots.txt crawler policy means for your website’s AI search visibility.
The short version
Perplexity says that PerplexityBot now crawls website content in compliance with robots.txt. If your robots.txt blocks PerplexityBot, Perplexity says it will not index the full or partial text content of those blocked pages. That makes your robots.txt policy an important technical setting for Perplexity search visibility.
What is PerplexityBot?
Perplexity describes PerplexityBot as a crawler designed to surface and link websites in Perplexity search results. According to its crawler documentation, PerplexityBot is not used to crawl content for AI foundation-model training.
Why this distinction matters
Allowing PerplexityBot is a search-visibility decision, not automatically a decision about granting access for model training. Your organization can evaluate Perplexity search discovery separately from the policies you use for other AI crawlers.
What changed?
Perplexity states that its crawler follows robots.txt directives and will not index the full or partial text content of pages that disallow PerplexityBot. It also says that its third-party crawling partners have been required to respect robots.txt, particularly for news publisher sites.
Important nuance
Perplexity says it may still index limited metadata for blocked pages, including the domain, headline, and a brief factual summary. Blocking PerplexityBot therefore restricts access to page text, but it may not remove every reference to the existence of a page or domain.
What this means for GEO
If you want public content to remain eligible to appear as a source in Perplexity search answers, first check whether PerplexityBot is unintentionally blocked. This is especially important after site migrations, CMS changes, security-rule updates, or copying a restrictive robots.txt template.
However, crawler access is only the first requirement. Allowing PerplexityBot does not guarantee that a page will be crawled, indexed, retrieved, cited, or recommended. Source selection can vary based on the user's question, competing pages, freshness, relevance, content clarity, and Perplexity's own systems.
Check your robots.txt rules
Open your site's robots.txt file at:
https://yourdomain.com/robots.txt
Look for a PerplexityBot-specific block, or a broad wildcard rule that might apply to it:
Blocks crawler access
User-agent: PerplexityBot Disallow: /
Allows public crawling
User-agent: PerplexityBot Allow: /
Only allow crawling if it matches your organization's privacy, licensing, security, and publishing policies. Sensitive areas such as admin panels, customer dashboards, staging sites, and private documents should remain protected by proper access controls.
Check more than robots.txt
A crawler may still have trouble accessing a page even when robots.txt allows it. Review these technical signals as well:
- Pages returning server errors, redirect loops, or inconsistent canonical URLs.
- Content hidden behind login walls, region gates, or aggressive bot-protection rules.
- JavaScript-heavy pages that return little useful HTML before scripts run.
- Important articles missing from your sitemap or internal linking structure.
- WAF or CDN rules that block legitimate crawler requests. Perplexity publishes IP ranges that can be used for verification and allowlisting where appropriate.
A practical checklist for Perplexity visibility
Review your robots.txt file for PerplexityBot blocks.
Confirm your important public pages are accessible without login.
Check that your key pages return meaningful HTML and have clear headings.
Use internal links and an XML sitemap to help discovery.
Keep your product, author, pricing, and factual information current.
Track a consistent group of Perplexity prompts and look for changes over time.
Free GazeRank tool
Check your AI crawler access
Use the GazeRank robots.txt Checker to see whether common AI crawlers, including PerplexityBot, appear allowed or blocked on your website.
Related learning