What is pay-per-crawl?
Pay-per-crawl is a model, offered through infrastructure providers like Cloudflare, that lets a site charge AI crawlers a fee for each page they fetch, turning crawler access from a binary allow-or-block decision in robots.txt into a priced transaction.
Traditional robots.txt only offers two states per crawler, allowed or disallowed. Pay-per-crawl adds a third: allowed, at a price. A site enrolled in this kind of program can set a per-request fee, and a crawler that wants to fetch a page either pays it or receives a blocked response instead, with the infrastructure provider handling billing between the two sides.
The tradeoff is that pricing a crawler out is functionally the same as blocking it. If a site sets its price high enough that no AI company chooses to pay, the practical outcome for citation is identical to a flat Disallow, even though the site technically left the door open. Whether pay-per-crawl helps or hurts visibility depends entirely on the price chosen and which crawlers actually pay it, not on the existence of the program itself.
Related
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
- AI crawlerAn AI crawler is a bot operated by an AI company to fetch web pages for training, indexing, or answering a live question.
- WAFA WAF, short for web application firewall, sits in front of a website and filters incoming traffic for threats, and its default bot-detection rules can silently block AI crawlers even when a site's robots.txt allows them, making it a common blind spot in access audits.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.