What is the Disallow directive?
Disallow is the line in robots.txt that tells a specific crawler which paths it should not request, and it works as a path prefix match rather than a keyword match, so Disallow: /blog blocks every URL starting with those characters, including ones an author never meant to hide.
Disallow is paired with a User-agent line above it and a path after it, and the crawler named in that User-agent line is asked to skip any URL beginning with that path. The common trip-up is treating it like a search term rather than a prefix: Disallow: /blog doesn't just block a page called /blog, it also blocks /blog-archive and /blogger-signup, anything starting with those characters, unless a trailing slash or wildcard narrows it.
Disallow only withholds permission to crawl going forward, it doesn't remove a page that's already indexed and it doesn't undo citations from a crawl that happened before the rule was added. Pulling an already-indexed page out requires a noindex tag that gets fetched while the page is still crawlable, or a direct removal request, since a page can't be told to update its own listing once it's no longer allowed to be visited.
Related
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
- the robots meta tagThe robots meta tag is an HTML tag placed in a page's head that tells a crawler what to do with that specific page after fetching it, most often noindex, and it only works if the crawler was allowed to fetch the page in the first place.
- the Allow directiveAllow is the robots.txt line that carves out an exception inside a broader Disallow rule, and when both match the same URL, the more specific rule wins regardless of which line comes first in the file.
- IndexationIndexation is whether a search engine has stored your page and is willing to return it. Being crawled is not the same as being indexed, and being indexed is not the same as ranking.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.