Are Fastly VCL rate limits dropping Perplexity-User requests?

You searched for your company on Perplexity, saw a competitor named instead, and decided to check your infrastructure. A look at your edge logs reveals a series of HTTP 429 Too Many Requests status codes served to requests carrying the Perplexity-User user-agent string. Fastly is dropping the connections before they ever reach your origin server.

To understand why this happens, you have to separate training crawlers from answer crawlers.

Training crawlers like GPTBot, ClaudeBot, and Google-Extended fetch web pages in bulk to build future model parameters. Blocking or rate limiting them affects whether your content enters a future model training cycle months from now. Answer crawlers like Perplexity-User, ChatGPT-User, Claude-SearchBot, and OAI-SearchBot operate on a completely different cycle. When a user submits a prompt, an answer engine sends an answer crawler to fetch relevant web pages in real time. The engine reads the retrieved text, synthesizes a response, and provides citations within seconds.

If your edge CDN drops an answer crawler, the AI engine does not wait for a retry. It drops your domain from consideration for that prompt immediately and uses the next available source.

How Fastly VCL Rate Limiting Triggers 429 Status Codes

Fastly uses Varnish Configuration Language (VCL) to process requests at the edge. Rate limiting in Fastly relies on VCL modules like rate.penaltybox alongside edge tables to track request volumes over rolling windows, such as 10 seconds or 60 seconds.

When a user asks an answer engine a complex question, the engine often issue requests for multiple pages on your domain in rapid succession. The answer crawler might hit your homepage, a feature page, and three documentation articles within 200 milliseconds to synthesize context.

Standard Fastly security configurations treat rapid sequential requests from shared edge IP addresses as synthetic web scraping or layer-7 denial-of-service attempts. A typical VCL rate-limiting implementation looks at request frequency per client IP or request fingerprint:

sub vcl_recv {
  # Block client IPs exceeding rate thresholds
  if (rate.penaltybox_has("ip_rate_limit", client.ip)) {
    error 429 "Too Many Requests";
  }

  if (req.url ~ "^/docs" && rate.check_rate("doc_requests", client.ip, 10s) > 20) {
    rate.penaltybox_add("ip_rate_limit", client.ip, 1m);
    error 429 "Too Many Requests";
  }
}

Because answer engines route thousands of real-time user queries through concentrated egress IP ranges, Perplexity-User traffic repeatedly trips these volume thresholds. The VCL policy pushes the egress IP into a penalty box and serves an HTTP 429 status code on subsequent requests.

In our crawler index, sampled across a panel of 54,082 domains, 33,670 domains returned a readable robots.txt file. Out of those readable domains, 2,576 explicitly block at least one answer crawler via robots.txt rules. Perplexity-User is explicitly blocked in robots.txt by 1,402 of 33,670 domains. However, rate-limiting drops happening inside Fastly VCL are silent blocks: your robots.txt allows the crawler, but your edge security drops the traffic before it touches your application layer.

Inspecting Fastly Log Streams for Perplexity-User

To verify whether Fastly is actively dropping Perplexity-User requests, you must inspect your streaming log endpoints. Fastly does not store raw request logs on the edge node; it streams them to external log management destinations such as Datadog, Amazon S3, Google BigQuery, or a custom syslog server.

First, check your VCL logging definitions to ensure that req.http.User-Agent and resp.status are captured in your log format strings. A standard Fastly custom log format snippet should include:

log {"{"}
  "timestamp": "" %{begin:%Y-%m-%dT%H:%M:%S%z}t "",
  "client_ip": "" %h "",
  "user_agent": "" %{User-Agent}i "",
  "request_url": "" %V%m %U%q "",
  "status": " %s ",
  "fastly_error": "" %{fastly_info.state}V ""
{"}"};

Once your log aggregator receives this stream, write a query to isolate traffic specifically from the Perplexity answer crawler. In Datadog, BigQuery, or ELK, filter your logs using this logic:

1. Filter for user_agent containing the case-sensitive string Perplexity-User.
2. Group the results by status code.
3. Compare the count of status 200 responses against status 429 or status 503 responses over a 7-day period.

If you see alternating patterns where a single request returns 200 followed immediately by three or four status 429 responses from the same client IP, your Fastly VCL rate limit is actively penalizing the crawler during multi-page context gathering.

Adjusting VCL Rules for Answer Crawlers

To resolve edge drops without disabling your protection against malicious web scrapers, modify your VCL rules in vcl_recv to handle identified answer crawlers separately.

You can create an exception or a higher rate-limit threshold specifically for user-agents corresponding to answer crawlers. Where possible, match the user-agent string while validating origin networks to prevent basic user-agent spoofing by malicious scrapers.

An updated VCL logic block allocates a higher request ceiling to recognized answer crawlers before invoking the penalty box:

sub vcl_recv {
  # Define higher rate thresholds for real-time answer crawlers
  if (req.http.User-Agent ~ "Perplexity-User|ChatGPT-User|Claude-SearchBot") {
    if (rate.check_rate("ai_answer_crawlers", client.ip, 10s) > 100) {
      rate.penaltybox_add("ip_rate_limit", client.ip, 30s);
      error 429 "Too Many Requests";
    }
  } else {
    # Standard security threshold for general traffic
    if (rate.check_rate("standard_traffic", client.ip, 10s) > 20) {
      rate.penaltybox_add("ip_rate_limit", client.ip, 1m);
      error 429 "Too Many Requests";
    }
  }
}

By increasing the allowed request volume within the 10-second window for answer crawlers, you permit Perplexity-User to fetch the three to six deep links it needs to render a cited answer without triggering a site-wide block.

What Edge Fixes Change (and What They Do Not)

Diagnosing and fixing rate limits requires understanding the limits of infrastructure changes.

First, log inspection and VCL edits require manual setup inside your CDN pipeline. No external tracking platform can automatically modify your Fastly service configurations or set up custom log streams on your behalf. You must implement log streaming and update your custom VCL files directly inside Fastly.

Second, fixing your Fastly VCL code to return HTTP 200 responses to Perplexity-User will not cause Perplexity or ChatGPT to immediately re-crawl your domain or cite your brand. Answer crawlers do not index pages on a fixed schedule like traditional search spiders. They run on demand when an end-user enters a prompt that the engine decides requires external web searching. If no user asks a query relevant to your domain, no answer crawler will visit, regardless of how clean your edge VCL policies are.

Finally, citation generation in AI engines is fundamentally non-deterministic. An engine can fetch your page with a 200 status code and still omit your brand from the final answer text due to prompt variability, competing source context, or model output decisions.

To measure whether your brand is cited over time, Standing evaluates citations across five engines: ChatGPT, Perplexity, Gemini, Claude, and Google AI Overviews. Instead of running a prompt a single time, our method uses a fixed prompt basket run five times per question per engine, calculating a Wilson interval on the results. A vendor that runs a prompt once might show 0 mentions today and 1 tomorrow, leading you to believe a Fastly VCL change fixed your visibility when you were simply seeing normal non-deterministic variance. Measuring 14, plus or minus 3 out of 25 runs provides an actual baseline for whether your infrastructure changes improved your brand's presence in answer engines.

We offer three tiers for tracking visibility across these engines: Track at $100 per month for 3 domains, Optimize at $300 per month for 5 domains with custom prompt baskets, and Agency at $500 per month for 50 domains with a monthly re-scan. Fixing your CDN rate limits clears the technical barrier so that answer crawlers can reach your content when a user prompt demands it.