Is Akamai Bot Manager blocking Claude-SearchBot requests?

Your robots.txt file permits Anthropic to crawl your site, but Claude refuses to cite your pages when users ask for product recommendations or real-time information. You search your standard Google Analytics dashboard and see zero recorded visits from Anthropic. The issue is almost certainly sitting at your CDN layer.

If your infrastructure relies on Akamai Bot Manager or Akamai Kona Site Defender, your edge network is likely dropping requests from Claude-SearchBot before those HTTP requests ever reach your application origin. Akamai is doing exactly what you hired it to do: protecting your origin from automated traffic. But because AI search engines rely on low-latency, automated HTTP clients to retrieve live web context, standard security scoring often misclassifies legitimate AI answer crawlers as malicious bot activity.

How Akamai Bot Manager Flags Claude-SearchBot Traffic

Akamai Bot Manager does not rely solely on declared User-Agent strings to classify traffic. If it did, any malicious scraper could bypass your firewall by setting its User-Agent to match a common search engine. Instead, Akamai uses automated behavioral scoring, TCP/IP stack fingerprinting, and TLS Client Hello analysis.

When a browser connects to your site, it presents a distinct TLS fingerprint: specific cipher suites, supported extensions, elliptic curve selections, and HTTP/2 frame settings. Standard web browsers like Chrome or Safari present fingerprints that Akamai recognizes instantly.

When Anthropic's real-time retrieval system requests a page, Claude-SearchBot initiates an outbound connection using a programmatic HTTP client library. Even though the request presents a legitimate User-Agent string identifying itself as Claude-SearchBot, its TLS client fingerprint reveals a non-browser stack.

Akamai calculates a bot score for every incoming request based on these factors:

1. Non-browser TLS fingerprints: The cipher suites presented do not match standard desktop or mobile web browsers.
2. Missing browser behaviors: The client does not evaluate JavaScript, download secondary CSS or image assets, or execute DOM rendering.
3. Network layer anomalies: Request pacing, TCP window sizing, and header ordering closely match automated scripting tools.

If your Akamai Bot Management policy is configured to block, challenge, or rate-limit traffic scored as high Anomaly or Automated, the edge drops the connection. Akamai responds directly to Claude-SearchBot with an HTTP 403 Forbidden or HTTP 428 Precondition Required status code. Because the connection terminates at the edge, your application server never logs the request, and your analytics script never fires.

Answer Crawlers Versus Training Crawlers at the Edge

A critical mistake teams make when configuring web application firewalls is treating all AI crawlers as a single category. Blocking a training crawler impacts whether your content is used to train future foundation models months from now. Blocking an answer crawler stops an AI engine from retrieving your live content right now.

Anthropic operates distinct agents for these separate tasks:

• ClaudeBot is a training crawler. It recursively collects web data to build training sets for future model generations.
• Claude-SearchBot and Claude-User are answer crawlers. When a human user asks Claude a question that requires current web search, Claude-SearchBot fetches relevant web pages in real time to summarize an answer and cite the source.

In our index of 54,082 domains, 33,670 returned a readable robots.txt. Within those 33,670 domains, 4,603 explicitly block ClaudeBot in their robots.txt rules. By contrast, only 1,400 of 33,670 block Claude-SearchBot. Across the full web panel, 5,497 domains block at least one AI agent, while 2,576 block at least one answer crawler.

In a separate census of 4,226 active Y Combinator companies conducted on 2026-08-07, 3,755 returned a readable robots.txt, and 253 of those companies blocked at least one AI crawler.

Many organizations intentionally block ClaudeBot to prevent training on their IP, while intending to allow Claude-SearchBot so they remain visible in live search responses. However, when Akamai Bot Manager triggers an automated TLS fingerprint block, it drops both agents indiscriminately. Your carefully configured robots.txt policy becomes completely irrelevant because the edge firewall terminates the connection long before robots.txt can be read.

Auditing Raw Akamai Edge Logs for HTTP 403 Status Codes

To confirm whether Akamai is actively blocking Claude-SearchBot, you must inspect raw edge logs. You cannot rely on origin access logs or application monitoring tools like DataDog or New Relic, as blocked requests never reach those systems.

Request your security or operations team to pull logs from Akamai DataStream 2 or Cloud Monitor for a 24-hour window. Filter the raw JSON log stream using the following criteria:

{
  "user_agent": "*Claude-SearchBot*",
  "response_status": 403
}

When analyzing the log entries, inspect the specific Akamai metadata fields tied to the request:

• edgeAction: Look for values like deny, abort, tarpit, or challenge. If this field indicates an action was taken while your origin log shows no corresponding request, Akamai intervened at the edge.
• botManagementScore or botScore: A score near 100 indicates Akamai scored the connection as highly automated.
• actionReason: Check if the engine flagged the request under BOT_MAN_DETECTED or TLS_FINGERPRINT_ANOMALY.

If you locate HTTP 403 entries carrying the Claude-SearchBot User-Agent alongside high bot scores, your firewall rules are actively blocking Anthropic's live search engine.

Creating Maintainable Exceptions Without Turn-Key Plugins

Fixing this block requires adding a explicit exception rule inside Akamai Control Center under your Kona Site Defender or Bot Manager configuration.

Do not attempt to fix this by purchasing turn-key bot management plugins, third-party edge middleware, or WordPress add-ons that promise automated AI search exemptions. These turn-key tools typically operate inside the application layer or rely on basic User-Agent string checking. They cannot override Akamai edge decisions, they introduce latency, and they often create security vulnerabilities by trusting unverified request headers.

The correct approach is to configure a native Akamai Custom Rule with dual-condition matching:

1. Request Header Condition: Match User-Agent containing Claude-SearchBot or Claude-User.
2. Network Condition: Match source IP addresses against Anthropic's published egress IP ranges.

You must require both conditions. Matching on User-Agent alone allows malicious scrapers to spoof the header and bypass your bot security entirely.

You must accept an operational reality when setting up this rule: whitelist rules must be manually maintained as crawler IP blocks change over time. Anthropic periodically updates, adds, or migrates the IP blocks used by its infrastructure. If your infrastructure team creates a static IP list in Akamai and never updates it, your firewall rule will quietly fail in six months when Anthropic provisions new edge nodes.

To maintain this properly, your engineering team must either subscribe to Anthropic's published IP list changes or write an automated worker that calls Anthropic's public IP endpoint and updates your Akamai Network Lists via the Akamai Open API.

Verifying Search Retrieval Changes

Once you update your Akamai rules to permit verified Claude-SearchBot traffic, you need to verify whether the change restored your visibility in Claude.

Do not test this by typing a single prompt into Claude and checking if your site appears. LLMs are non-deterministic. A single prompt response is a sample size of one and contains high statistical noise. An engine might skip your page on one run due to context window limits, then cite it on the next run.

To reliably measure whether your site is being recommended, you must run prompt testing systematically across multiple iterations. At Standing, we measure brand visibility using a fixed prompt basket. We run each prompt five times per question across five major engines: ChatGPT, Perplexity, Gemini, Claude, and Google AI Overviews. We then calculate a Wilson interval on the aggregate results. A metric like "34, plus or minus 6" provides an actual statistical measurement of your visibility, whereas a single pass or fail test tells you almost nothing about your true standing.

If your Akamai edge logs show zero 403 errors for Claude-SearchBot across a 48-hour window, and your multi-run prompt sampling demonstrates consistent citations, your firewall configuration is complete.