What is crawl-delay?
Crawl-delay is a robots.txt line that asks a crawler to wait a set number of seconds between requests, but it was never part of the original robots.txt standard, and most major AI crawlers ignore it entirely.
Crawl-delay looks like a standard robots.txt directive, Crawl-delay: 10 asks a bot to wait ten seconds between hits, but it was invented by individual search engines rather than the protocol itself, and support has always been inconsistent. Google has never honored it for Googlebot, and most AI crawlers make no public commitment to respect it either, so adding it is a request, not a control.
A site genuinely worried about crawler load, AI or otherwise, is usually better served by server-side rate limiting or a bot-management layer that can actually throttle or reject requests, since those enforce the limit rather than ask for it. Crawl-delay is worth setting for the handful of crawlers that do check it, but it shouldn't be the only defense against a bot hitting a server too hard.
Related
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
- AI crawlerAn AI crawler is a bot operated by an AI company to fetch web pages for training, indexing, or answering a live question.
- Crawl budgetCrawl budget is how much crawling a search engine is willing to spend on your site. On a new or low-authority domain it is small, which is why publishing more pages can reduce how many get indexed.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.