What is a WAF?

A WAF's job is security, catching scrapers, credential-stuffing attempts, and malicious traffic before it reaches the server. Many WAFs ship with bot-detection rules that flag any automated, non-browser traffic as suspicious by default, and an AI crawler making rapid, scripted requests can look exactly like the traffic those rules were built to stop, even when it's behaving exactly as its own documentation says it will.

This is why checking robots.txt alone doesn't confirm a crawler can actually reach a site. A site can explicitly allow GPTBot in robots.txt and still return a blocked or challenge response at the WAF layer, because the two systems are unrelated and neither one knows what the other decided. Confirming real access means checking server logs or a live fetch, not just reading the robots.txt file.