Why ChatGPT recommends your competitor instead of you

Someone on your team ran a prompt in ChatGPT asking for the top software in your category. Your primary competitor was listed second. Your company was not listed at all.

The instinct is to treat this as an SEO failure or a missing meta tag. It is usually neither. When an AI engine generates an answer, it selects information using a pipeline that behaves very differently from traditional search engines. It fetches pages live, extracts facts from plain text, or summarizes third-party sites that it already trusts.

When ChatGPT leaves you out, one of three things is happening: the answer crawler cannot fetch your site, your page text cannot be parsed into clear claims, or the engine is relying on listicles where you do not appear.

Cause 1: Answer crawlers cannot reach your site

Before ChatGPT can quote your site in a live response, its answer crawler must be allowed to read your pages. Many engineering teams update robots.txt rules to block AI scrapers without realizing they are blocking live retrieval at the same time.

You must distinguish between training crawlers and answer crawlers. A training crawler collects web pages to train future foundation models. Blocking a training crawler like GPTBot, ClaudeBot, or Google-Extended stops an AI company from using your content in its next core model update, but it does not stop current web search citations.

An answer crawler runs live when a user submits a prompt. If you block an answer crawler, ChatGPT cannot fetch your domain during a search-augmented query.

In our crawler index of 54,082 domains, 33,670 domains published a readable robots.txt file. Out of those 33,670 domains, 5,497 blocked at least one AI agent, and 2,576 blocked at least one answer crawler.

The distinction matters because vendors often blur these agents together. Our data shows that 5,080 out of 33,670 domains block GPTBot, and 4,603 out of 33,670 block ClaudeBot, both of which are training crawlers. Meanwhile, 2,074 out of 33,670 domains block ChatGPT-User, 1,579 out of 33,670 block OAI-SearchBot, 1,402 out of 33,670 block Perplexity-User, and 1,400 out of 33,670 block Claude-SearchBot. These four are answer crawlers. If your site blocks ChatGPT-User or OAI-SearchBot, ChatGPT cannot pull live information from your domain today.

The pattern holds across startup ecosystems as well. In a census of 4,226 active Y Combinator companies, 3,755 had readable robots.txt files, and 253 blocked at least one AI crawler.

How to test for retrieval issues

Open your domain's robots.txt file in a browser by visiting your main URL ending in /robots.txt. Look for user-agent rules disallowing any of these agents:

• ChatGPT-User
• OAI-SearchBot
• Perplexity-User
• Claude-SearchBot

If any of these agents are set to Disallow: /, your server is explicitly rejecting live search requests from that engine.

Next, check your Web Application Firewall or Cloudflare settings. Security defaults often block unrecognized automated bots. Check your server access logs for HTTP 403 or 429 status codes linked to the user-agent ChatGPT-User. If your firewall drops those requests, ChatGPT treats your site as unreachable and falls back on your competitor.

Cause 2: Your pages are fetchable but not quotable

If ChatGPT can reach your site, it parses the rendered text to extract factual statements. Large language models do not read pages visually. They process raw text streams.

A page that relies heavily on slogans, text rendered inside graphics, or complex JavaScript frameworks can yield empty text to an answer crawler. If your competitor's page explicitly states what their product does in simple declarative sentences, the model extracts their claims and skips yours.

Do not expect schema markup or an llms.txt file to fix this. Schema markup provides key-value pairs for structured database parsers, and an llms.txt file is simply a Markdown directory listing helpful page links. Neither acts as a direct ranking signal or guarantees that an engine will cite you. Mechanically, an llms.txt file only gives a crawler a clean index of plain-text URLs to traverse. It cannot force a model to select your brand over another.

How to test for quotability

Copy the raw, unformatted text of your main product page and paste it into a plain text editor. Strip out all layout, design, and navigation elements.

Read the plain text and evaluate three things:

1. Does the text state your product category in the first two sentences?
2. Are your key features, pricing terms, and target users named directly as factual statements?
3. Can a reader extract a clear sentence about your product without context from surrounding images or icons?

If your site requires a human to interpret visual layout or watch a video to understand what you sell, an answer crawler will fail to construct a reliable quote. Your competitor gets recommended because their copy is easily restated by a text prediction model.

Cause 3: You are missing from cited third-party sources

When ChatGPT answers a prompt like "What are the best tools for X?", it rarely starts by guessing company URLs and inspecting individual homepages. It executes a traditional search query behind the scenes, fetches top web results, and synthesizes the consensus across those pages.

For most commercial prompts, top search results are listicles, review aggregators, and comparison blogs. If three separate software review sites list your competitor as a top choice for a workflow, ChatGPT adopts that consensus. If your brand is absent from those external pages, ChatGPT will not invent a recommendation for you, regardless of how clear your own website copy is.

Fixing this issue takes time and happens mostly off your site. Editing your homepage, adding internal blog posts, or modifying site metadata will not fix a lack of third-party coverage.

How to test for third-party presence

Run your target prompt directly in ChatGPT with web browsing enabled. Look at the inline citations and source links listed in the footnotes of the response.

List every domain cited in that output. Visit those specific URLs and search for your brand name.

If your competitor appears in four of those cited sources and your company appears in zero, ChatGPT is accurately summarizing the external web consensus it retrieved. Your work here is not technical site optimization, but securing placement on the industry sites that rank for those background search queries.

How to measure your recommendations correctly

AI engines are non-deterministic. A prompt run at nine in the morning can return a different list of brands than the exact same prompt run five minutes later.

Because of this variance, a single manual search tells you very little. If you run a query once and see your brand, you cannot assume you are consistently recommended. If you run it once and miss, you cannot assume you are invisible.

Vendor tools that run a prompt once and report a simple yes or no hide this underlying variance. Stating a single score without a range is meaningless. A real measurement requires running the same prompt multiple times and calculating a confidence band.

At Standing, we track domains across five engines: ChatGPT, Perplexity, Gemini, Claude, and Google AI Overviews. We run a fixed prompt basket five times per prompt per engine and report results using a Wilson interval. A real measurement looks like a recommendation score of 34, plus or minus 6, calculated at a 95% confidence interval. Competitor tools run each prompt once, which is why none of them publish a statistical band.

When evaluating vendor offerings, avoid PR packages promising guaranteed AI mentions, automated schema injectors, or tools claiming to write llms.txt files that force ChatGPT to rank you. AI engines cannot be forced into deterministic outputs. Focus on unblocking answer crawlers, writing explicit plain-text claims on your domain, and earning placement in the third-party lists that engines read during live retrieval.