How many sites block AI crawlers
Published estimates for this range from 3.5% to 88.9%. They are not really in conflict. Each one measured a different set of websites and then got quoted without it. Here is ours, and here is the denominator it depends on.
Out of the 1597 whose robots.txt we could establish, not out of the 2698 we asked.
What we threw out
1101 of 2698 domains refused us their robots.txt or failed to answer. They are excluded rather than counted as unblocked.
A rule we were never shown looks exactly like no rule at all. Counting those in the clear would push the headline down by however many strict firewalls the sample happened to contain, and sites that run one are not a random subset.
By crawler
Every count is out of the same 1597 domains. Blocking is a per-agent decision and reporting it per agent is the only way to see the finding below.
| Crawler | Feeds | Domains | Share |
|---|---|---|---|
| GPTBot | ChatGPT | 266 | 16.7% |
| ClaudeBot | Claude | 241 | 15.1% |
| Google-Extended | Gemini | 222 | 13.9% |
| ChatGPT-User | ChatGPT | 129 | 8.1% |
| PerplexityBot | Perplexity | 126 | 7.9% |
| OAI-SearchBot | ChatGPT | 101 | 6.3% |
| Claude-SearchBot | Claude | 96 | 6.0% |
| Claude-User | Claude | 92 | 5.8% |
| Perplexity-User | Perplexity | 92 | 5.8% |
| Googlebot | Gemini / AI Overviews | 39 | 2.4% |
The interesting part is not the headline
GPTBot is disallowed on 266 domains. OAI-SearchBot on 101. Both are OpenAI. GPTBot trains and grounds the models; OAI-SearchBot builds the index ChatGPT cites from when it answers a question.
So roughly 62% of the sites refusing OpenAI are refusing to be trained on while remaining entirely citable. That is a deliberate distinction and, as far as we can tell, a correct one.
It also explains a result that otherwise looks like a paradox. BuzzStream found that 88.2% of news sites blocking GPTBot still get cited by AI. If most blockers never blocked the search crawler, there is nothing to explain.
Which means the sentence you will read everywhere, including in cold emails, is wrong for most of the sites it gets said to. “GPTBot is blocked, so ChatGPT cannot see you” is only true if OAI-SearchBot is blocked too. Check which one before you act.
Method
The panel. Tranco daily list Q2X24 (generated 2026-08-05), ranks 2000-50000, excluding hostnames matching a fixed infrastructure/CDN/ad-tech pattern, then a systematic every-9th sample of the 47,341 eligible rows. Frozen: membership never changes. New domains belong to a new panel with its own manifest.
What counts as readable. A 2xx we could parse, or a 404 or 410. A 404 is an answer: the server replied and there is no such file, so nothing is disallowed there. A 403, a timeout or a 5xx is not an answer, and those domains are excluded.
What we did not do. We did not treat a server refusing our live request as a block. We send a crawler's User-Agent from our own IP, which is what an impersonator looks like, so a bot-protection service refusing us is that service working correctly rather than evidence about the crawler.
Run. Completed August 6, 2026, from a single vantage point, one moment in time. The panel is frozen and will be re-run on the same domains, so the next edition is a change rather than a new sample.
What this does not show
The 40.8% exclusion rate is the biggest weakness here and it is not random. Sites behind strict firewalls are dropped systematically, and if those sites block AI crawlers at a different rate than the ones that let us read, the headline is off by an amount we cannot measure from here.
robots.txt is also a request rather than a wall. It tells you what a site asks crawlers to do, not what any crawler did. And access is not citation: a crawler permitted to read a page may never fetch it, and one that fetches it may never quote it.
We publish this weakness because the alternative is a cleaner number that means less. If you want to check any of it, the crawler checker below runs the same measurement on one domain and shows the responsible robots.txt line.
Get the next edition
The panel is frozen, so the same domains get re-checked and the next edition reports what moved rather than a new sample. One email when it runs, and nothing else.
Run this on your own domain
The same check, one site at a time, free and without an account. It reports the exact robots.txt line responsible and separates a real block from a firewall that simply refused us.