Is GPTBot blocked on your site? How to check, and what it costs you

If an AI crawler can't fetch your pages, nothing else you do about AI visibility matters. No amount of good content, structured data or brand mentions will get you named in an answer built from sources the engine was never allowed to read.

This is the cheapest thing to check and the most expensive thing to get wrong, so it belongs first.

The crawlers that matter, and who they feed

There are more than most people expect, because each engine runs several with different jobs.

Agent	Engine	What it does
GPTBot	ChatGPT	Trains and grounds OpenAI models
OAI-SearchBot	ChatGPT	Builds ChatGPT's search index
ChatGPT-User	ChatGPT	Fetches a page live when a user's question needs it
ClaudeBot	Claude	Trains and grounds Anthropic models
Claude-User	Claude	Fetches a page during a conversation
Claude-SearchBot	Claude	Builds Claude's search index
PerplexityBot	Perplexity	Builds Perplexity's index
Perplexity-User	Perplexity	Fetches a page to answer a live question
Google-Extended	Gemini	Controls Gemini grounding and training
Googlebot	Gemini, AI Overviews	Feeds Google's index, including AI Overviews

The split between index bots and user bots is the part people miss. Blocking GPTBot but allowing ChatGPT-User means you stay out of the index but can still be fetched when someone asks a question that names you specifically. Blocking both means neither.

The Google-Extended question

This is the one that causes the most confusion, so it is worth being precise.

Google-Extended controls whether your content is used for Gemini grounding and training. It does not affect your Google Search ranking. You can disallow it and rank exactly as before in Search.

Googlebot is the one that does both. It feeds the Search index, and AI Overviews are built on that index. So disallowing Googlebot removes you from Google Search and from AI Overviews together, which is almost never what anyone wants.

Blocking GPTBot, ClaudeBot or PerplexityBot has no effect on Google Search at all. They're different crawlers run by different companies.

robots.txt is only half the answer

Here's the failure mode that catches people who have already checked their robots file.

robots.txt is a request, not a wall. It's a published preference that well-behaved crawlers honour voluntarily. It does not prevent a request from being made, and it has no bearing on how your server responds.

So there are two ways to be blocked:

1. In robots.txt. A Disallow rule matching the agent. Visible if you read the file.
2. At the server or CDN. A WAF rule, bot-protection setting or rate limit that returns 401, 403 or 429 to anything without a browser fingerprint. Completely invisible in robots.txt.

The second is more common than people expect, and it's usually nobody's decision. It arrives as a default in a security product, or as a "block bad bots" toggle somebody enabled two years ago. Cloudflare, Akamai, AWS WAF and most managed hosts all ship rules that can catch AI crawlers in a net meant for scrapers.

The only way to find it is to make an actual request using the crawler's own User-Agent and look at the status code. Reading the file will never show you.

How to check

Manually, for one agent:

curl -sI -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" https://yoursite.com

A 200 means you're reachable. A 401, 403 or 429 means the edge is refusing, regardless of what robots.txt says. Repeat for each agent in the table above, because rules are often written per-agent.

Then read robots.txt and check for rules matching each one. Watch for the pattern where several agents share a block:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Disallow: /

Consecutive User-agent lines share the rules that follow. It's easy to read that as one rule for PerplexityBot and miss that it applies to all three.

If you'd rather not do it by hand, our AI crawler access checker does both halves for all ten agents in about three seconds. It's free, needs no signup, and quotes the responsible robots.txt line when robots is the cause.

What a clean result does and doesn't mean

If every crawler comes back allowed, you have removed one specific obstacle. That's genuinely worth doing, and it's the only fix on the list you can finish in ten minutes.

It does not mean an engine has read you, and it certainly doesn't mean one will recommend you. Access is permission to fetch. Being named in an answer depends on what the engine found when it did, and on what it has read about you elsewhere, which is usually third-party sources rather than your own site.

Treat this as the floor. It's the thing you check so that when the number is still low, you know it isn't this.

If you find something blocked

In robots.txt: decide deliberately. Some publishers block AI crawlers on purpose, because they don't want their content used for training without payment. That's a legitimate position with a real cost: it also removes you from the answers those engines give. If you're trying to be recommended, allow them.

At the server: find the rule. It'll be in your CDN's bot-management settings, your WAF rules, or your host's security defaults. Add the AI agents to the allow list. Check again afterwards, because these settings frequently have a cache and a "verified bot" category that behaves differently from a manual rule.

Either way, re-check after the change. The gap between "we updated the config" and "the crawler can actually get in" is where most of these problems live.