What is Bytespider?
Bytespider is ByteDance's web crawler, used both to gather training data for its AI models and to fetch pages for Doubao, its AI chat product, which means one user agent currently serves the same dual role that GPTBot and OAI-SearchBot split into two at OpenAI.
Bytespider is ByteDance's crawler, the company behind TikTok and the AI assistant Doubao. Unlike OpenAI, which runs a separate training crawler, GPTBot, and a separate answer-fetching crawler, OAI-SearchBot, ByteDance has used Bytespider for both jobs under one name, so a site can't currently allow it for one purpose and block it for the other the way it can with OpenAI's bots.
Bytespider has also drawn attention for crawling more aggressively than most bots of its kind, which is a separate problem from whether to allow it at all. A site worried about server load from Bytespider should look at rate limiting or its hosting provider's bot management settings rather than robots.txt alone, since robots.txt only asks a crawler to slow down or stay away, it can't enforce it.
Related
- GPTBotGPTBot is OpenAI's web crawler. It reads pages to train and ground OpenAI's models, including ChatGPT.
- OAI-SearchBotOAI-SearchBot is the crawler OpenAI uses to build the index ChatGPT cites from. It is not the training crawler, and blocking it is a different decision from blocking GPTBot.
- AI crawlerAn AI crawler is a bot operated by an AI company to fetch web pages for training, indexing, or answering a live question.
- crawl-delayCrawl-delay is a robots.txt line that asks a crawler to wait a set number of seconds between requests, but it was never part of the original robots.txt standard, and most major AI crawlers ignore it entirely.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.