What is X-Robots-Tag?
X-Robots-Tag does the same job as the robots meta tag, noindex and similar instructions, but delivers it as part of the server's response rather than inside the page's HTML, which makes it the only option for files that have no HTML to put a tag in, like PDFs or images.
A meta tag only works inside HTML, so a page's PDF version, an image, or a downloadable spreadsheet has no place to hold one. X-Robots-Tag solves that by sending the same instruction, noindex being the most common, as part of the server's response headers instead, which a crawler reads whether the file is HTML or not.
Because it lives in server configuration rather than in a page template, X-Robots-Tag is also the more reliable way to apply a rule across an entire file type at once, for instance adding noindex to every URL ending in .pdf without editing each file individually. It's invisible to a normal site visitor and to most page-inspection habits, so it's an easy place for a stale noindex to sit unnoticed for years after the reason for it is gone.
Related
- the robots meta tagThe robots meta tag is an HTML tag placed in a page's head that tells a crawler what to do with that specific page after fetching it, most often noindex, and it only works if the crawler was allowed to fetch the page in the first place.
- the Disallow directiveDisallow is the line in robots.txt that tells a specific crawler which paths it should not request, and it works as a path prefix match rather than a keyword match, so Disallow: /blog blocks every URL starting with those characters, including ones an author never meant to hide.
- IndexationIndexation is whether a search engine has stored your page and is willing to return it. Being crawled is not the same as being indexed, and being indexed is not the same as ranking.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.