What is ClaudeBot?
ClaudeBot is Anthropic's training crawler. Claude-SearchBot and Claude-User are separate agents that affect whether Claude can cite you.
Anthropic runs three named agents and they do different jobs. ClaudeBot collects data for training. Claude-SearchBot builds the index Claude uses for search. Claude-User fetches a page live when a conversation needs it. Blocking all three is a coherent position; blocking only the first is the more common one.
In our index ClaudeBot is disallowed on 4,603 of 33,670 readable domains, against 1,400 for Claude-SearchBot and 1,384 for Claude-User. The same three-to-one pattern as OpenAI, and for the same reason: most site owners object to being trained on rather than to being quoted.
Related
- GPTBotGPTBot is OpenAI's web crawler. It reads pages to train and ground OpenAI's models, including ChatGPT.
- OAI-SearchBotOAI-SearchBot is the crawler OpenAI uses to build the index ChatGPT cites from. It is not the training crawler, and blocking it is a different decision from blocking GPTBot.
- AI crawlerAn AI crawler is a bot operated by an AI company to fetch web pages for training, indexing, or answering a live question.
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.