What is the robots meta tag?
The robots meta tag is an HTML tag placed in a page's head that tells a crawler what to do with that specific page after fetching it, most often noindex, and it only works if the crawler was allowed to fetch the page in the first place.
The robots meta tag, written as a meta element with name robots and a value like noindex, sits inside a page's own HTML rather than in robots.txt. It gives per-page instructions, noindex to keep a page out of search results, nofollow to withhold endorsement from its outbound links, and it applies only to the page it's placed on, unlike robots.txt, which applies to whole path patterns from one central file.
The catch is sequencing. A crawler has to fetch a page to read its meta tag, so a page blocked in robots.txt is never fetched, which means its noindex tag is never seen and has no effect. This trips up site owners trying to remove an already-indexed page: adding noindex works only if the page stays crawlable, and blocking it in robots.txt at the same time freezes it in whatever state the last successful crawl left it in.
Related
- the Disallow directiveDisallow is the line in robots.txt that tells a specific crawler which paths it should not request, and it works as a path prefix match rather than a keyword match, so Disallow: /blog blocks every URL starting with those characters, including ones an author never meant to hide.
- X-Robots-TagX-Robots-Tag does the same job as the robots meta tag, noindex and similar instructions, but delivers it as part of the server's response rather than inside the page's HTML, which makes it the only option for files that have no HTML to put a tag in, like PDFs or images.
- IndexationIndexation is whether a search engine has stored your page and is willing to return it. Being crawled is not the same as being indexed, and being indexed is not the same as ranking.
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.