Understanding llms.txt: The New Standard for AI Crawlers
What is llms.txt, how does it differ from robots.txt and a sitemap, and when is it worth adding a curated AI-readable guide to your website?
The honest 2026 status
llms.txt is an open proposal and a useful convention for publishing a curated map of content. It is not a replacement for robots.txt, is not a web standard in the same sense as robots.txt or sitemaps, and does not guarantee crawling, rankings, or AI citations. Google explicitly says it does not use llms.txt for Google Search or its generative AI features.
What is llms.txt?
An llms.txt file is a Markdown document, usually published at /llms.txt, that introduces a site or section and links to its most important resources. The proposal is designed to give language models and agents a concise starting point instead of making them infer a site’s structure from navigation, templates, and large collections of pages.
It is most natural for product documentation, API references, knowledge bases, and other sites with a well-defined set of authoritative pages. A good file is short, maintained, and selective. Think of it as a curated reading list—not a full site inventory.
llms.txt vs. robots.txt vs. sitemap.xml
| File | Primary purpose | What it does not do |
|---|---|---|
| robots.txt | Communicates crawl access preferences to participating bots. | It is not a content summary or a citation signal. |
| sitemap.xml | Lists URLs you want search engines to discover and consider. | It does not explain which pages are most useful for a topic. |
| llms.txt | Offers a curated, Markdown-readable overview and links. | It does not control bot access or guarantee visibility. |
What actually affects AI discovery?
Start with fundamentals. Make key pages public and crawlable, use a sound technical structure, publish accurate and genuinely useful information, and make it easy for people and crawlers to navigate between related content. Use robots.txt to manage a crawler’s access only when that crawler documents support for those directives.
For example, OpenAI documents OAI-SearchBot as the crawler used to surface websites in ChatGPT search. If appearing in those search results is important to you, review that crawler’s official guidance and your robots.txt configuration. Adding llms.txt alone does not grant permission or establish that your content will appear in an answer.
Use robots.txt for crawl controls
Robots.txt communicates which automated crawlers may access paths on your site. It is the relevant place to allow or disallow participating crawlers, such as OpenAI's OAI-SearchBot.
Use sitemap.xml for discovery
Your XML sitemap should list canonical, indexable URLs you want search engines to discover. Keep it current and submit it through the relevant webmaster tools.
Use llms.txt as an optional content guide
An llms.txt file can provide a small, curated map of your most useful resources—especially documentation. It is advisory content, not an access-control file or a confirmed ranking signal.
A practical llms.txt example
The format is deliberately simple: a site name, a brief summary, and Markdown lists of curated links. Replace the example URLs and descriptions with pages that represent your current canonical information.
# GazeRank
> GazeRank helps teams monitor website health, technical SEO, and AI search visibility.
Use these resources to understand our product and learn how to improve a website's readiness for search and AI systems.
## Product
- [Website Health Scanner](https://example.com/): Scan a domain for performance, SEO, accessibility, and security issues.
- [AI Visibility](https://example.com/ai-visibility): Measure whether a brand appears across supported AI search experiences.
## Learn
- [GEO Essentials](https://example.com/learn/geo): Guides to AI crawlers, citations, structured data, and AI-friendly content.
- [Core Web Vitals](https://example.com/blog/core-web-vitals): A practical guide to LCP, INP, and CLS.Implementation checklist
- Publish the file at /llms.txt, or at a relevant subpath when it describes only that section of the site.
- Start with one H1 containing the organization, project, or product name.
- Add a short blockquote summary explaining what the site or section is for.
- Group a small set of high-value canonical URLs under clear H2 headings.
- Write a factual, useful description for every linked resource.
- Use stable, absolute HTTPS URLs and verify that every link returns the intended page.
- Keep robots.txt, XML sitemaps, canonical tags, and on-page content accurate; llms.txt does not replace them.
Should you add one?
It can be a low-cost addition when you have documentation or a knowledge base and can maintain a concise, accurate file. It may make your content easier for compatible agents to orient themselves around. Do not treat it as a priority over pages that are broken, blocked from crawling, duplicate, outdated, or unhelpful.
The strongest approach is to publish it as a helpful supplement while continuing to invest in the fundamentals that are documented to matter: unique, people-first content, crawlable pages, clear internal structure, and crawler-specific access controls where applicable.
Next step
Check your AI search readiness
Scan your domain with GazeRank to find crawlability, content, structured-data, performance, and accessibility issues that can limit discoverability.
Scan a website →