What is Common Crawl?
Common Crawl is a nonprofit that has published free, periodic snapshots of large parts of the web since 2008, and its dataset, fetched by a crawler called CCBot, has become a default ingredient in training data for many language models, including some built by companies that never crawled your site themselves.
Common Crawl isn't an AI company. It's a nonprofit that runs its own crawler, CCBot, and publishes the results as petabyte-scale archives that anyone can download for free. Dozens of research labs and AI companies use these archives as a training data source because building a comparable crawl from scratch is expensive, so blocking CCBot in robots.txt is really a decision about all the downstream models that might train on that archive, not just one company's bot.
The part people miss is the lag. Common Crawl publishes monthly snapshots and keeps every past one, so content it already crawled sits in archives that a robots.txt change today cannot pull back. A site that blocks CCBot going forward stops contributing to future snapshots, but a model trained last year on an older snapshot was already built with that page included, and no amount of blocking un-trains it.
Related
- GPTBotGPTBot is OpenAI's web crawler. It reads pages to train and ground OpenAI's models, including ChatGPT.
- Google-ExtendedGoogle-Extended is a robots.txt token that controls whether your content trains Gemini and grounds its answers. It is not a crawler and blocking it does not affect Google Search ranking.
- Training vs search crawlersA training crawler collects data models learn from. A search crawler builds the index an assistant cites when answering. Blocking one has nothing to do with the other.
- AI crawlerAn AI crawler is a bot operated by an AI company to fetch web pages for training, indexing, or answering a live question.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.