How to read your own robots.txt for AI crawlers

You typed a prompt into ChatGPT asking for vendor recommendations in your category, and your brand was missing. Someone on your team opened yourdomain.com/robots.txt, saw a list of Disallow directives, and declared that your technical setup is blocking AI engines from seeing your company.

Before you ask engineering to edit the file or pay a consultant to fix your technical SEO, you need to read what the file actually says. Most teams misread their own directives because they treat every crawler with "GPT" or "Bot" in its name as the same entity.

The distinction that breaks most audits

The single most common error in reading a robots.txt file is confusing a training crawler with an answer crawler. They perform entirely different jobs for an AI company, and blocking one has a completely different effect than blocking the other.

A training crawler collects web content to build future base models. If you block a training crawler today, it has zero impact on whether ChatGPT or Perplexity cites your site right now. The model already finished training on historical data months ago.

An answer crawler fetches live web pages while generating a response for a user. If an engine decides to search the web to answer a prompt, the answer crawler executes that HTTP request. Blocking an answer crawler stops the engine from reading your page in real time.

In our index of 54,082 domains, 33,670 returned a readable robots.txt file. Across those 33,670 domains, 5,497 block at least one AI crawler. However, only 2,576 block at least one answer crawler. More than half of the sites that believe they are blocking AI access have only blocked training bots. Their live answer visibility is completely unrestricted by their robots.txt file.

Reading the agent lines in your file

A robots.txt file works through user-agent blocks. Each block specifies an agent name followed by path rules like Allow or Disallow. Here is how to evaluate each major AI agent you might find in your configuration.

OpenAI agents

OpenAI uses distinct user-agent strings for training, system operations, and live web search:

• GPTBot is the primary training crawler. In our data set, 5,080 of 33,670 readable domains block GPTBot. Blocking this agent prevents OpenAI from using your content in future model training runs. It does not stop ChatGPT from browsing your site during a live user query.
• ChatGPT-User is an answer crawler used when a user directly prompts ChatGPT to browse or when the system needs to fetch a link provided in a prompt. It is blocked by 2,074 of 33,670 readable domains.
• OAI-SearchBot is the dedicated prototype search crawler used by OpenAI's search features. It is blocked by 1,579 of 33,670 readable domains.

If your file blocks GPTBot but does not block ChatGPT-User or OAI-SearchBot, ChatGPT can still visit your domain during live searches.

Anthropic agents

Anthropic similarly splits its infrastructure across training and live interaction:

• ClaudeBot is the training crawler. It is blocked by 4,603 of 33,670 readable domains. Blocking it stops content accumulation for future Claude models.
• Claude-SearchBot is the web search crawler used to retrieve current information for user queries. It is blocked by 1,400 of 33,670 readable domains.
• Claude-User executes requests initiated directly by a user within a Claude session. It is blocked by 1,384 of 33,670 readable domains.

Perplexity agents

Perplexity relies heavily on real-time retrieval:

• PerplexityBot acts as a general indexer and search crawler for Perplexity. It is blocked by 2,147 of 33,670 readable domains.
• Perplexity-User carries out specific, user-triggered retrieval tasks during conversational search. It is blocked by 1,402 of 33,670 readable domains.

Google agents

Google maintains a strict separation between traditional search indexing and AI model training:

• Googlebot is the standard search indexer. It is blocked by 610 of 33,670 readable domains. Blocking Googlebot removes you from traditional Google Search as well as AI Overviews.
• Google-Extended is a specific control that allows site owners to opt out of Gemini training and Google's generative AI model development without affecting standard search ranking. It is blocked by 4,275 of 33,670 readable domains.

Path rules, inherited blocks, and ghost rules

Reading the user-agent name is only the first step. You must also check the path attached to the directive and where the rule originated.

Wildcard traps

If your file contains a wildcard block like this:

User-agent: *
Disallow: /

Every AI crawler that respects robots.txt will stop crawling your site immediately, including answer crawlers. Unless an agent is explicitly allowed with its own dedicated section higher in the file, a global Disallow: / applies to all bots.

Path scope

A Disallow rule only applies to the specific URL path matching the string. For example:

User-agent: ChatGPT-User
Disallow: /api/
Disallow: /staging/

This configuration blocks ChatGPT-User from accessing your API endpoints and staging environment, but leaves your marketing pages, blog posts, and product documentation fully open. If your brand is missing from a ChatGPT output, this rule is not the cause. The engine was never going to fetch your /api/ path to answer a user recommendation prompt.

Security products and CDN defaults

Many robots.txt rules were never written by a developer on your team. Cloudflare, AWS CloudFront, Imperva, and other content delivery networks or security proxies offer single-toggle switches to block AI crawlers.

In some cases, these edge networks intercept the bot at the firewall level and return a 403 Forbidden HTTP status code before the bot ever reaches your server or reads your robots.txt file. In other cases, the CDN automatically injects Disallow directives into your served robots.txt file.

To test this, fetch your robots.txt using an external terminal command like curl -A "ChatGPT-User" https://yourdomain.com/robots.txt rather than relying on the source file in your code repository.

This pattern is widespread across high-growth technology companies. In our census of 4,226 active Y Combinator companies, 3,755 returned a readable robots.txt file, and 253 blocked at least one AI crawler. In many cases, these blocks were added automatically by security middleware during initial domain setup.

What fixing robots.txt will and will not do

If you discover that your site blocks answer crawlers like ChatGPT-User or PerplexityBot, removing those disallow lines will allow those engines to fetch your pages live during search operations.

However, unblocking an answer crawler does not guarantee that an engine will mention your brand. AI engines are non-deterministic systems. They choose which sites to pull based on prompt relevance, query intent, source domain authority, and existing model training.

Do not purchase tools or services that promise guaranteed AI citations, and do not spend money on software claiming to format your site specifically for LLMs. There is no reliable evidence that adding schema markup or publishing an llms.txt file increases citation rates or improves recommendations in generative engines.

If you want to evaluate whether your brand is actually recommended by AI engines, you must measure citations accurately across multiple runs. Generative models produce different responses to identical inputs. Testing a prompt once in a web browser tells you nothing about your true visibility.

At Standing, we track recommendations across five engines: ChatGPT, Perplexity, Gemini, Claude, and Google AI Overviews. We execute a fixed prompt basket five times per question per engine and calculate the result using a Wilson interval, presenting visibility as a range such as 34, plus or minus 6. Competitor tools typically run a prompt once, which provides a single static number but hides the underlying variance of the model.

Start by verifying your robots.txt file with a text editor. Ensure your answer crawlers are open, leave training bots to your legal or policy preference, and evaluate your AI visibility using statistical sampling rather than single-prompt assumptions.