What is PerplexityBot?
PerplexityBot builds Perplexity's index. Unlike OpenAI and Anthropic, Perplexity runs no separate training crawler, so every block against it affects citation.
This makes Perplexity the outlier and it changes how a block should be read. With OpenAI or Anthropic, a robots.txt rule is usually a training opt-out with citation left intact. With Perplexity there is no training-only option: PerplexityBot and Perplexity-User both serve retrieval, so disallowing either is a decision to be absent from answers.
Our index finds 2,147 domains disallowing PerplexityBot and 1,402 disallowing Perplexity-User, out of 33,670 readable. Every one of those is a citation block, which is why Perplexity's blocked-share looks smaller than OpenAI's while mattering more per case.
Related
- AI crawlerAn AI crawler is a bot operated by an AI company to fetch web pages for training, indexing, or answering a live question.
- CitationA citation is a source an AI assistant links or refers to in support of an answer it has given.
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.