The AI Crawler Access Index
How much of the web refuses to be read by AI, measured rather than estimated. Published figures for this range from 5.89% to 26.15%, which is not a disagreement about the web so much as four different populations measured once each. This one is a fixed panel, re-run, with the raw data attached.
Out of the 33,670 whose robots.txt we could establish, never out of the 54,082 we asked.
Only half of that affects citation
2,921 of 5,497 blocking domains disallow training crawlers only. Those sites stay entirely quotable, and a tool reporting them as invisible to AI is wrong about 53% of the cases it finds.
Why every published figure disagrees
These are not competing estimates of one quantity. Each measured a different set of websites, which is the whole explanation, and none of them travel with that set attached when they get quoted.
| Source | Figure | Population measured |
|---|---|---|
| Originality.ai | 26.15% | top 1,000 websites |
| Standing | 16.3% | 33,670 domains with a readable robots.txt |
| Cloudflare | 11.0% | 3,973 popular domains |
| Ahrefs | 5.89% | 140M sites |
Ours is not the right one. It is the one whose population, exclusions and raw rows are published, so the figure can be recomputed against a different denominator by anyone who prefers theirs.
By engine
Blocking is not one decision. Refusing to be trained on and refusing to be quoted are different choices, and most sites making the first are not making the second.
| Engine | Blocks anything | Training only | Affects citation |
|---|---|---|---|
| ChatGPT | 5,144 | 5,080 | 2,180 |
| Claude | 4,629 | 4,603 | 1,457 |
| Perplexity | 2,187 | 0 | 2,187 |
| Gemini / AI Overviews | 4,297 | 4,275 | 0 |
GPTBot is disallowed 3.2× more often than OAI-SearchBot. Perplexity is the outlier in the other direction: it runs no separate training crawler, so every block against it is a citation block.
Every agent
- GPTBottraining5,08015.1%
- ClaudeBottraining4,60313.7%
- Google-Extendedtraining4,27512.7%
- PerplexityBotsearch2,1476.4%
- ChatGPT-Usersearch2,0746.2%
- OAI-SearchBotsearch1,5794.7%
- Perplexity-Usersearch1,4024.2%
- Claude-SearchBotsearch1,4004.2%
- Claude-Usersearch1,3844.1%
- Googlebotother6101.8%
Take the data
Published under CC BY 4.0. If you disagree with the denominator, recompute it: every domain we could not establish is in the file and flagged rather than dropped.
Method, and what this does not show
How. One HTTPS request to /robots.txt per domain, parsed for Disallow rules applying to each of ten named AI user-agents. No JavaScript, no login, nothing that cannot be reproduced with curl.
The denominator. A 2xx we could parse counts. A 404 counts, because it is an answer: there is no file, so nothing is disallowed. A 403, a timeout or a TLS failure does not, and 20,412 domains fall in that bucket. Counting those as unblocked would drag the figure down by however many strict firewalls the panel happens to contain, and sites running one are not a random subset.
What it is not. Not a measure of whether anyone gets cited. robots.txt is a request, and a blocked crawler is not an absent one. Content an engine learned before a rule was added does not disappear, and other sites writing about a company stay readable. Anyone claiming a robots.txt line makes a site invisible to AI is overreading a text file.
Check a single domain
The same check runs on any site, free and without an account, and reports training and search crawlers separately rather than collapsing them into one count.