Does Vercel Edge Middleware block AI search crawlers by default?

Someone told you your brand does not show up in ChatGPT. You run on Vercel, and now you are wondering if your Edge Middleware is silently rejecting the bots fetching your site.

The short answer is no. Vercel Edge Middleware does not contain a default rule that blocks AI search crawlers out of the box.

The longer answer is that the custom code running inside your middleware very well might.

How Edge Middleware Misroutes AI Crawlers

Edge Middleware executes code at the network edge before a request reaches your page rendering logic or static cache. Founders and engineering teams commonly add middleware to handle geolocation redirects, basic authentication, bot detection, or security headers.

When an AI search engine attempts to retrieve your page to answer a user prompt, it sends an HTTP request with a specific User-Agent header. If your middleware checks User-Agent strings against a strict allowlist, or if it uses a third-party bot detection library designed to challenge unknown headless browsers, the request gets caught.

Instead of serving the page HTML, your edge function might return a 403 Forbidden status, a 307 temporary redirect to a login screen, or a blank fallback layout. To the AI agent, your site appears down, private, or non-existent.

Training Crawlers vs. Answer Crawlers

Before inspecting your code, you must separate two entirely different types of AI crawlers: training crawlers and answer crawlers. Confusing these two is the most common operational mistake teams make when inspecting server logs.

Training crawlers collect web data to build future foundational models. Block them today, and your site content will not be included in model iterations released months or years from now. Blocking a training crawler does not prevent an engine from retrieving your live URL during an active search query if the engine uses a separate real-time search bot.

Answer crawlers execute live web searches to answer a user query right now. If your middleware blocks an answer crawler, the engine cannot retrieve your page content to synthesize an answer today.

Here is the explicit breakdown of these agents:

• Training crawlers: GPTBot, ClaudeBot, Google-Extended.
• Answer crawlers: ChatGPT-User, OAI-SearchBot, Perplexity-User, PerplexityBot, Claude-SearchBot, Claude-User.

In our crawler index run on 2026-08-08, we inspected 54,082 domains, of which 33,670 published a readable robots.txt file. Out of those 33,670 readable domains, 5,497 block at least one AI agent, and 2,576 block at least one answer crawler.

Looking specifically at answer crawlers, 2,074 of 33,670 domains block ChatGPT-User, 1,579 block OAI-SearchBot, 2,147 block PerplexityBot, 1,402 block Perplexity-User, 1,400 block Claude-SearchBot, and 1,384 block Claude-User. For context, standard web crawler Googlebot is blocked by 610 of 33,670 domains.

Even among venture-backed startups, crawler blocking is widespread. In our Y Combinator census executed on 2026-08-07 across 4,226 active Y Combinator companies, 3,755 had readable files, and 253 blocked at least one AI crawler.

Diagnosing Middleware with cURL

Because edge middleware executes before application logging in many setups, your primary analytics dashboard might never register these blocked requests. You must test your edge deployment directly using command-line requests that mimic these exact bot signatures.

Run cURL against your production domain using the User-Agent string of an answer crawler. For example, to test how your edge middleware handles OpenAI search retrieval:

curl -i -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" https://yourdomain.com

Repeat this check for ChatGPT-User:

curl -i -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)" https://yourdomain.com

And for Perplexity:

curl -i -A "Perplexity-User/1.0 (+https://www.perplexity.ai/perplexity-user)" https://yourdomain.com

Inspect the HTTP status code and response headers returned in the terminal output.

If your terminal receives a 200 OK status along with your full document payload, your edge middleware is passing that crawler through cleanly.

If your terminal receives a 403 Forbidden, 401 Unauthorized, 429 Too Many Requests, or a 307 Temporary Redirect header pointing to a login or blocked route, your edge middleware logic is actively blocking real-time AI retrieval.

Check your middleware implementation for conditional checks that evaluate request headers. If you use geo-routing based on request geolocation headers, ensure that requests originating from data centers where AI engines host their retrieval infrastructure are not being force-redirected to internationalized subpaths that lack your core product copy.

What Fixing Your Middleware Will and Will Not Do

If you find a bug in your edge middleware and deploy a patch, you have cleared the pipe. But you must manage your expectations about what happens next.

Fixing a broken middleware condition takes developer time, and it will not magically cause ChatGPT, Perplexity, Claude, Gemini, or Google AI Overviews to immediately recommend your business.

AI search engines are non-deterministic systems. They do not maintain an instant index update pipeline the way traditional keyword search engines do. Clearing a blocked User-Agent header simply means that when an engine decides to crawl your site to answer a query, your server will respond with content rather than an error code.

Do not pay an agency that promises to guarantee citations or instantly inject your company into AI answers after fixing a crawler block. Vendors promising guaranteed AI rankings are selling modern snake oil.

Similarly, do not waste engineering hours adding schema markup or creating an llms.txt file in the hope of forcing visibility. There is no empirical evidence that adding schema markup or llms.txt improves AI citations, and structured markup does nothing to fix an edge function returning a 403 status code.

Measuring Presence Reliably

Once your edge infrastructure allows answer crawlers through, how do you know if your brand is actually being recommended?

You cannot rely on running a single prompt in ChatGPT once on your laptop. AI engines output varied responses based on model non-determinism, conversation context, and minor temperature shifts. A single query that mentions your brand is not a measurement.

To measure brand visibility accurately, you must run prompt baskets repeatedly across engines to establish a statistical range.

At Standing, our approach uses a fixed prompt basket run five times per question per engine across five covered engines: ChatGPT, Perplexity, Gemini, Claude, and Google AI Overviews. We calculate a Wilson interval on the results.

When you view a visibility score in Standing, you see a statistical range: a brand recommendation frequency expressed as a count with a margin of error, such as 34, plus or minus 6. Competitors in this space run each prompt only once, which is why none of them publishes a confidence band.

Our plans reflect this focus on accurate, recurring tracking: Track costs $100 per month for 3 domains. Optimize costs $300 per month for 5 domains with custom prompts. Agency costs $500 per month for 50 domains with a monthly re-scan.

Fixing your Vercel Edge Middleware is a prerequisite for live retrieval, not a magic fix for visibility. Start by running cURL requests to verify your status codes, strip out overly restrictive User-Agent filters, and monitor your actual retrieval rates over time using repeated testing.