Which AI crawler blocks actually cost you citations

Your engineering team added a block for GPTBot to your robots.txt file last quarter to protect your IP from being used in AI training sets. Today, your marketing team noticed that your product is missing from ChatGPT recommendations when buyers ask for tools in your category.

It is easy to assume those two events are directly connected. Vendors selling bot-blocking software often tell you that blocking AI scrapers keeps your site private, while AI optimization platforms imply that unblocking every bot will magically land you citations.

Neither statement reflects how these systems actually work. The reality depends entirely on which specific user-agent you blocked, because AI companies maintain entirely separate crawler fleets for model training and real-time retrieval.

The two distinct jobs of AI crawlers

To understand why a block hurts or helps your visibility, you must split AI web crawlers into two distinct functional categories: training crawlers and answer crawlers.

Training crawlers fetch content to build the dataset for the next generation of foundation models. OpenAI uses GPTBot for this purpose. Anthropic uses ClaudeBot. Google uses Google-Extended. These crawlers run asynchronously, scanning millions of web pages months before a model goes into production.

Blocking a training crawler affects whether your site enters the model's static weights for future base releases. It has zero impact on whether an engine can visit your site to cite it today.

Answer crawlers fetch content on demand while a user is actively typing a prompt. When someone asks ChatGPT to recommend a software vendor, ChatGPT uses an answer crawler to perform a live web search, retrieve current pages, and synthesize an answer with links.

OpenAI uses ChatGPT-User and OAI-SearchBot for real-time search. Anthropic uses Claude-SearchBot and Claude-User. Perplexity uses PerplexityBot and Perplexity-User.

If you block an answer crawler, the engine cannot inspect your site during a live retrieval augmented generation run. The engine will fail to read your landing page, fall back on other sources, and cite your competitors instead.

Auditing your robots.txt line by line

Most robots.txt files are written once and forgotten. To know whether your current configuration is harming your live visibility, you must inspect the specific User-agent headers defined in your server directives.

A directive aimed at a training bot looks like this:

User-agent: GPTBot
Disallow: /

This instruction tells OpenAI not to scrape your site to build future base models. It does not stop ChatGPT from visiting your website during a live user query, because live queries do not use GPTBot.

Conversely, a directive aimed at an answer bot looks like this:

User-agent: ChatGPT-User
Disallow: /

This instruction actively prevents ChatGPT from following a link or inspecting your domain while generating a response for a user right now. Blocking ChatGPT-User stops real-time citations immediately.

If your robots.txt contains a wildcard disallow rule like this:

User-agent: *
Disallow: /

You are blocking every answer crawler alongside traditional search engine spiders, unless you explicitly grant access to specific user agents later in the file.

What the web crawler data reveals

Many companies block training crawlers without realizing they are turning off answer crawlers, or they block answer crawlers by accident when trying to limit aggressive scrapers.

We analyzed a panel of 54,082 domains on August 8, 2026. Out of those domains, 33,670 returned a readable robots.txt file. Within those 33,670 readable domains, 5,497 blocked at least one AI agent, but only 2,576 blocked at least one answer crawler.

The breakdown across the 33,670 readable domains shows how heavily skewed site configurations are toward blocking training bots rather than search bots:

• GPTBot (training): 5,080 domains blocked
• ClaudeBot (training): 4,603 domains blocked
• Google-Extended (training): 4,275 domains blocked
• PerplexityBot (search): 2,147 domains blocked
• ChatGPT-User (search): 2,074 domains blocked
• OAI-SearchBot (search): 1,579 domains blocked
• Perplexity-User (search): 1,402 domains blocked
• Claude-SearchBot (search): 1,400 domains blocked
• Claude-User (search): 1,384 domains blocked
• Googlebot (other): 610 domains blocked

The numbers confirm that site owners block training bots at more than double the rate of answer bots. Blocking GPTBot or ClaudeBot stops an AI provider from training future model weights on your content, but it does not remove your site from live search results today. Live citations are only cut off when you block answer crawlers such as ChatGPT-User, OAI-SearchBot, or Claude-SearchBot.

A separate census of active Y Combinator companies conducted on August 7, 2026, looked at 4,226 domains. Of the 3,755 domains with readable robots.txt files, 253 blocked at least one AI crawler.

In many of these cases, startups inadvertently blocked answer crawlers because they activated default security rules inside their edge network or web application firewall.

Edge security rules and third-party tools

If you use edge security networks like Cloudflare or specialized bot-management vendors, you may be blocking answer crawlers without any mention of them in your robots.txt file.

Cloudflare provides a single toggle to block AI bots at the edge. If your primary corporate objective is minimizing server load or stopping unauthorized scraping of proprietary assets, Cloudflare's native firewall rules are far more effective and easier to maintain than hand-coded web server directives.

However, edge management tools often bundle training crawlers and answer crawlers into a single broad security category. When you flip a switch to block AI scrapers, the edge network drops connections from ChatGPT-User and Perplexity-User at the network layer before those bots ever read your robots.txt file.

Do not buy security software that promises to selectively hide your content from AI scrapers while promising that search engines will still index you perfectly. Every rule you implement at the firewall layer introduces a trade-off between infrastructure protection and real-time search discovery.

Do not buy agency packages that promise guaranteed citations, mentions, or rankings in AI engines once you unblock these bots. AI engines are non-deterministic systems. Unblocking an answer crawler gives the engine the technical ability to read your site, but it does not force the model to prefer your site over a competitor.

Measuring visibility without guessing

Because AI models generate responses dynamically, testing a single prompt once in ChatGPT or Perplexity tells you very little. An engine might cite your site on one run and omit it on the next due to standard model variance, even if your site is fully crawlable.

Single-run testing tools cannot separate an active crawler block from a temporary API timeout or a non-deterministic response shift.

At Standing, we run a fixed prompt basket five times per question across five major engines: ChatGPT, Perplexity, Gemini, Claude, and Google AI Overviews. Because model outputs fluctuate, we calculate scores using a Wilson confidence interval rather than publishing single point estimates.

A domain score might be reported as 42, plus or minus 5, across a sample of buyers asking for recommendations in your niche. If you block answer crawlers like ChatGPT-User or Perplexity-User, that score will drop near zero for live search prompts on those platforms, because the model simply cannot inspect your pages to verify its answer.

We offer three tiers for tracking these metrics: Track at $100 per month for 3 domains, Optimize at $300 per month for 5 domains with custom prompts, and Agency at $500 per month for 50 domains with a monthly re-scan.

If your brand is absent from AI answers, start by inspecting your web server configuration and edge network settings. Ensure your firewalls distinguish between offline training crawlers and live answer crawlers before you change your content strategy.