What is a user agent?
A user agent is the name a client sends to identify itself when requesting a page. robots.txt rules are written against these names, which is why blocking AI crawlers means naming each one.
There is no wildcard that means every AI crawler. A rule targeting one name does nothing to another, so a robots.txt that disallows GPTBot leaves ClaudeBot, PerplexityBot and the rest untouched. This is the most common reason a site owner believes they have opted out and has not.
User agents are also self-reported and trivially forged. A request claiming to be GPTBot may be anyone, which is why the operators publish IP ranges and support reverse-DNS verification. Anything you conclude from your logs about crawler behaviour is only as reliable as the verification behind it.
Related
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
- AI crawlerAn AI crawler is a bot operated by an AI company to fetch web pages for training, indexing, or answering a live question.
- Bot verificationBot verification is confirming that a request claiming to come from a named crawler really did, using reverse DNS or the operator's published IP ranges rather than trusting the user-agent string.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.