Is an AI assistant allowed to fetch your site at all?
Assistants do not all read the web with one crawler. There are 10 named agents in common use, they arrive under different user agents, and a site can allow some and refuse others without anyone intending it. The refusals usually come from a rule written years ago for a search crawler that a newer agent now matches. A source an engine cannot fetch cannot be a source it quotes, so this is the check that has to pass before any other one means anything.
- Check
- 1 of 6
- Needs
- Your domain
- Cost
- Free
Do this on your own site
A free tool. It needs your domain and nothing else.
Enter your domain. It asks each named agent, one at a time, whether your server lets it in, and shows the rule responsible when the answer is no.
AI Crawler Access CheckerReading the result
Including what it does not tell you, which is the half most readings get wrong.
A blocked agent is a fact about your server, not a verdict on your visibility, and the two get confused constantly. Being reachable is a precondition, not an achievement: a site every crawler can read can still go unmentioned in every answer, which is what the later lessons measure. Note also that the agents split into two jobs, training and search, and blocking a training crawler is a decision about model training rather than about whether you are named.
Terms used in this lesson
Each one is defined in full in the glossary, with the reason it matters to measurement.
Next: Readable answer
Does the answer exist in what your server actually sends?