Is AWS WAF silently blocking Perplexity-User requests?

You search Perplexity for a query where your product should rank, but your domain is completely absent. Your engineering team checks robots.txt and confirms there is no rule blocking Perplexity. The file is clean, valid, and permits access. Yet when Perplexity tries to synthesize an answer about your category, it skips your site and cites your closest competitors instead.

The issue is rarely in your robots.txt. If your application sits behind AWS Web Application Firewall (WAF), default managed rule sets are likely intercepting real-time requests from Perplexity and dropping them with an HTTP 403 Forbidden status code before the traffic ever hits your application origin or reaches your robots.txt parser.

How AWS WAF managed rules block live search crawlers

To understand why AWS WAF drops these requests, you must distinguish between training crawlers and answer crawlers.

Training crawlers, such as GPTBot, ClaudeBot, and Google-Extended, scan the web asynchronously to build datasets for future foundation models. Blocking a training crawler prevents your text from entering the next model training run, but it has zero impact on whether an AI engine can cite your site today.

Answer crawlers execute in real time when a user asks a question. When someone enters a prompt into Perplexity, the engine dispatches Perplexity-User to fetch live pages from the web to cite in its response. ChatGPT-User, Claude-SearchBot, OAI-SearchBot, and Claude-User perform the exact same function for their respective platforms.

AWS WAF managed rules inspect inbound HTTP requests against automated signatures to protect servers from distributed denial of service attacks, credential stuffing, and web scraping. Two managed rule groups in particular frequently flag Perplexity-User:

1. AWSManagedRulesCommonRuleSet: Contains the NoUserAgent_Header and UserAgent_BadBots_Header rules. These rules evaluate the string structure of the incoming User-Agent header. Custom or non-browser user-agent strings that do not match standard desktop browser signatures are routinely categorized as suspicious scrapers.
2. AWSManagedRulesBotControlRuleSet: Evaluates client JA3 fingerprints, IP reputation, and request frequency. Because Perplexity-User routes requests dynamically through cloud hosting providers to fetch page content quickly, its request profile matches automated headless scraper patterns.

When an answer crawler like Perplexity-User triggers these rules, AWS WAF executes a Block action at the Amazon CloudFront or Application Load Balancer edge. The request terminates immediately with an HTTP 403 response. Perplexity receives an empty or forbidden response, marks your site as unreachable for that prompt execution, and falls back to a competitor whose server returns an HTTP 200 OK.

Diagnosing a silent WAF block using curl and logs

You do not need specialized software to verify whether your firewall is rejecting Perplexity-User requests. You can test your site directly from your command line.

Run a curl request against a high-value URL on your domain while overriding the request header with Perplexity's live fetch user-agent:

curl -i -A "Perplexity-User" https://yourdomain.com/product-page

Look at the first line of the HTTP response headers:

• If the response is HTTP/2 200 or HTTP/1.1 200 OK, your firewall is permitting the user-agent string through to your web server.
• If the response is HTTP/2 403 or HTTP/1.1 403 Forbidden, your firewall or edge platform is actively rejecting the user-agent.

If the command yields a 403 error, confirm that AWS WAF is the source by examining your WAF logged events in Amazon CloudWatch or S3. Filter your WAF log stream for the following field values:

{
  "httpRequest.headers[user-agent]": "Perplexity-User"
}

Inspect the terminatingRuleId field in the log entry. If the request was blocked by WAF, you will see entries citing rules such as AWSManagedRulesCommonRuleSet or AWSManagedRulesBotControlRuleSet. The log explicitly records the drop action, proving that the block occurs at the firewall level rather than inside your application logic.

Fixing the block in AWS WAF

Consultants and agencies frequently charge thousands of dollars to audit "AI search visibility" when the underlying problem is a single incorrect firewall rule. You do not need an agency or expensive security advisory to resolve an AWS WAF block.

To allow Perplexity-User through AWS WAF, your infrastructure team simply needs to add an explicit override rule to your Web ACL.

1. Navigate to the AWS WAF console and select your Web ACL.
2. Add a custom rule placed above your AWS Managed Rule groups in the rule evaluation order.
3. Configure the statement to inspect the User-Agent header for the string Perplexity-User.
4. Set the action to Allow (or Count if you want to inspect traffic volumes before fully allowing it).

By placing this explicit rule higher in priority than AWSManagedRulesCommonRuleSet or AWSManagedRulesBotControlRuleSet, AWS WAF approves the matching request before the general bot detection rules evaluate it.

At Standing, our platform monitors these edge block events across AI engines. We track visibility by running a fixed prompt basket five times per question per engine, calculating the result with a Wilson interval to account for variance. Our platform costs $100 per month for Track (3 domains), $300 per month for Optimize (5 domains with custom prompts), and $500 per month for Agency (50 domains with a monthly re-scan).

When our system runs checks across ChatGPT, Perplexity, Gemini, Claude, and Google AI Overviews, Standing detects HTTP 403 response codes returned to answer crawlers. Standing can identify that your server returned a forbidden status code to Perplexity-User, but Standing cannot automatically alter your AWS WAF configuration rules. The technical fix remains entirely inside your own AWS console.

The wider landscape of crawler blocking

Firewall blocks are often invisible because companies focus exclusively on robots.txt. In our crawler index updated on August 8, 2026, we analyzed a panel of 54,082 domains. Of those, 33,670 domains returned a readable robots.txt file.

Within those 33,670 readable domains, 5,497 block at least one AI agent in robots.txt, and 2,576 block at least one answer crawler. Looking specifically at individual agents blocked via robots.txt:

• GPTBot (training): 5,080 of 33,670 domains
• ClaudeBot (training): 4,603 of 33,670 domains
• Google-Extended (training): 4,275 of 33,670 domains
• PerplexityBot (search indexer): 2,147 of 33,670 domains
• ChatGPT-User (answer crawler): 2,074 of 33,670 domains
• OAI-SearchBot (answer crawler): 1,579 of 33,670 domains
• Perplexity-User (answer crawler): 1,402 of 33,670 domains
• Claude-SearchBot (answer crawler): 1,400 of 33,670 domains
• Claude-User (answer crawler): 1,384 of 33,670 domains
• Googlebot (general crawler): 610 of 33,670 domains

These robots.txt figures capture explicit, intentional blocks written by site owners. They do not account for the additional domains where Perplexity-User is blocked accidentally by AWS WAF, Cloudflare Bot Management, or Akamai Bot Manager before robots.txt is ever fetched.

A census of 4,226 active Y Combinator companies conducted on August 7, 2026, showed that 3,755 had readable robots.txt files, and 253 blocked at least one AI crawler explicitly. Yet many founders in that dataset who intend to be accessible to AI engines remain invisible simply because their cloud infrastructure drops answer crawlers by default.

If your content is missing from Perplexity responses, do not start by rewriting your content or buying schema markup packages. Verify your server's edge responses first using curl. If AWS WAF is returning a 403, write an exception rule for Perplexity-User. Once the firewall permits the fetch, the AI engine can pull your content in real time.