What is an AI crawler?
An AI crawler is a bot operated by an AI company to fetch web pages for training, indexing, or answering a live question.
Each major engine runs several with different jobs, and the split matters: index crawlers build a corpus ahead of time, while user-triggered crawlers fetch a page during a conversation. Allowing one and not the other produces partial visibility that is hard to diagnose from a score alone.
Access can be denied in two places. robots.txt is the visible one; a CDN or firewall rule returning 403 to anything without a browser fingerprint is the one that catches people who have already checked their robots file.
Related
- GPTBotGPTBot is OpenAI's web crawler. It reads pages to train and ground OpenAI's models, including ChatGPT.
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
- CitationA citation is a source an AI assistant links or refers to in support of an answer it has given.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.