What is the Allow directive?
Allow is the robots.txt line that carves out an exception inside a broader Disallow rule, and when both match the same URL, the more specific rule wins regardless of which line comes first in the file.
Allow exists for cases like blocking an entire folder except one page inside it: Disallow: /members/ followed by Allow: /members/directory lets a crawler reach that one directory page while staying out of everything else under /members/. Without Allow, there'd be no way to punch a hole in a broader block short of restructuring the URLs themselves.
The rule that resolves conflicts between Disallow and Allow is specificity, the longest matching path wins, not the order the lines appear in the file. Someone who lists Allow: /members/directory above Disallow: /members/ and assumes the Allow wins because it comes first has the right outcome by accident; if the paths were reversed in length, listing Allow first wouldn't save it. Checking a live testing tool against the actual rule, rather than trusting file order, is the only reliable way to confirm which one applies.
Related
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
- the Disallow directiveDisallow is the line in robots.txt that tells a specific crawler which paths it should not request, and it works as a path prefix match rather than a keyword match, so Disallow: /blog blocks every URL starting with those characters, including ones an author never meant to hide.
- Crawl budgetCrawl budget is how much crawling a search engine is willing to spend on your site. On a new or low-authority domain it is small, which is why publishing more pages can reduce how many get indexed.
- User agentA user agent is the name a client sends to identify itself when requesting a page. robots.txt rules are written against these names, which is why blocking AI crawlers means naming each one.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.