What is content licensing?
Content licensing is a paid agreement that gives an AI company the right to use a publisher's material, typically for training a model, in exchange for money or other consideration.
These deals sit apart from the everyday mechanics of crawling. A crawler reading a public page and a licensing agreement are two different relationships: one is unpaid access under terms set by robots.txt, the other is a negotiated contract, often covering back catalogs the crawler could never reach on its own, like paywalled archives.
The detail people miss is that a licensing deal usually governs training data, not what happens when a user asks a live question. A publisher can be paid to have its archive folded into a model's training set and still be effectively invisible when that same model does a live, retrieval-based answer, if its current site blocks the separate crawler used for that purpose. The two are governed by different mechanisms and often by different bots entirely.
Related
- Training vs search crawlersA training crawler collects data models learn from. A search crawler builds the index an assistant cites when answering. Blocking one has nothing to do with the other.
- Copyright and AI trainingThis is the unresolved legal question of whether training a model on copyrighted text or images, without a license, counts as infringement, and it is currently being fought out in courts rather than settled by any clear rule.
- GPTBotGPTBot is OpenAI's web crawler. It reads pages to train and ground OpenAI's models, including ChatGPT.
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.