What is robots.txt?
robots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
Well-behaved crawlers honour it voluntarily. It does not stop a request being made and has no bearing on how your server responds, which is why a site can be blocking AI crawlers at its CDN while its robots.txt says everything is allowed.
For AI visibility the relevant entries are the AI-specific agents, and they need to be read carefully: consecutive User-agent lines share the rules that follow them, so one Disallow can apply to several bots at once.
Related
Want to know where you actually stand on this? Run a free visibility check or try the free tools.