What is opt-out?
Opting out is telling an AI crawler, usually through robots.txt, not to access your site, and it works only if that crawler chooses to honor the request.
The word suggests a formal, enforceable choice, like unsubscribing from an email list. In practice it is a request written into a text file that a well-behaved crawler checks before it fetches a page. Nothing stops a crawler from ignoring it, and a site has no way to confirm compliance beyond watching server logs for that bot's traffic afterward.
Opting out is also crawler-specific and forward-looking only. Blocking one bot does nothing to another company's crawler, and it does nothing to content already collected in a past crawl, sold through a licensing deal, or mirrored on a site the publisher does not control. A decision to opt out today changes what gets fetched tomorrow, not what a model already learned yesterday.
Related
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
- AI crawlerAn AI crawler is a bot operated by an AI company to fetch web pages for training, indexing, or answering a live question.
- Bot verificationBot verification is confirming that a request claiming to come from a named crawler really did, using reverse DNS or the operator's published IP ranges rather than trusting the user-agent string.
- The text and data mining exceptionThe text and data mining exception is a copyright law provision, most developed in the EU and UK, that allows automated analysis of copyrighted works without separate permission, for specific purposes and subject to rights holders opting out.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.