Robots.txt, Meta Robots, and X-Robots-Tag

Why this matters

Robots.txt controls whether a crawler fetches a URL at all; meta robots and X-Robots-Tag control what happens after a page is fetched, such as whether it's indexed or how it can be presented in results. A page can be crawlable but noindexed, or blocked from crawling entirely regardless of its meta tags, and mixing these up leads to contradictory configurations.

A particularly common failure mode is blocking a page in robots.txt while also relying on a noindex meta tag to remove it from search — since the crawler never fetches the page, it never sees the noindex directive, and the page can remain indexed from before the block was added, with no way to signal removal.

Implementation guidance

Treat the three mechanisms as a layered system and check them together, not independently.

1. 1

   Map out the intended state for each priority page

   Decide explicitly whether each key page should be crawlable, indexable, and presentable in full, and document that intent.

2. 2

   Never combine a robots.txt block with a noindex tag for the same URL

   If you want a page removed from the index, allow it to be crawled so the noindex directive can actually be seen and processed.

3. 3

   Check X-Robots-Tag HTTP headers, not just meta tags

   Headers set at the server or CDN level can silently override or duplicate what's in the page's HTML meta tags.

4. 4

   Audit for leftover staging directives

   Confirm no noindex or nosnippet directives meant for a staging environment shipped to production pages.

5. 5

   Re-verify after any CDN or server configuration change

   Header-level robots directives are easy to introduce accidentally through infrastructure changes that never touch the codebase.

Validation checklist

• No priority page combines a robots.txt block with a noindex tag.
• X-Robots-Tag HTTP headers have been checked, not just in-page meta tags.
• No staging-only noindex or nosnippet directives are present in production.
• Directives were re-verified after the most recent CDN or server config change.