What is bot verification?
Bot verification is confirming that a request claiming to come from a named crawler really did, using reverse DNS or the operator's published IP ranges rather than trusting the user-agent string.
The user-agent header is a claim, not evidence. Scrapers routinely impersonate GPTBot or Googlebot to get past rules written for them, so log analysis that counts user-agent strings is counting assertions.
The check is straightforward. Take the requesting IP, do a reverse DNS lookup, confirm the hostname belongs to the operator's domain, then forward-resolve that hostname and confirm it returns the original IP. OpenAI, Anthropic, Perplexity and Google all publish IP ranges as an alternative.
This also explains a confusing result in access checks: a live probe refused by a CDN is not proof the real crawler is blocked. A request carrying a crawler's name from an unrelated address is exactly what an impersonator looks like, so the refusal may be the protection working correctly.
Related
- User agentA user agent is the name a client sends to identify itself when requesting a page. robots.txt rules are written against these names, which is why blocking AI crawlers means naming each one.
- AI crawlerAn AI crawler is a bot operated by an AI company to fetch web pages for training, indexing, or answering a live question.
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.