What is OAI-SearchBot?
OAI-SearchBot is the crawler OpenAI uses to build the index ChatGPT cites from. It is not the training crawler, and blocking it is a different decision from blocking GPTBot.
This is the single most consequential distinction in AI crawler access and almost every checker on the market collapses it. GPTBot gathers data models are trained on. OAI-SearchBot builds the search index ChatGPT draws on when it answers a live question with sources attached. A site can refuse the first and remain entirely quotable.
Our own index puts numbers on the gap: across 33,670 domains with a readable robots.txt, GPTBot is disallowed on 5,080 and OAI-SearchBot on 1,579, a ratio of roughly three to one. Of every domain blocking anything, 53% block only training crawlers and stay fully citable.
The practical consequence: if a tool tells you a robots.txt line has made you invisible to AI, check which agent. More than half the time that claim is wrong.
Related
- GPTBotGPTBot is OpenAI's web crawler. It reads pages to train and ground OpenAI's models, including ChatGPT.
- AI crawlerAn AI crawler is a bot operated by an AI company to fetch web pages for training, indexing, or answering a live question.
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
- CitationA citation is a source an AI assistant links or refers to in support of an answer it has given.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.