Which sites block AI crawlers

We read robots.txt on 200 of the most visited sites on the web and recorded which AI crawlers each one disallows. Every page publishes the rule responsible and the date it was taken, so any of it can be checked in a few seconds.

The distinction that matters throughout: blocking a training crawler keeps a site out of the models. Blocking a search crawler affects whether an assistant can cite it. 40 of these sites do the second; 4 do only the first.