Why AI answer crawlers fail on infinite scroll pagination

When you open an ecommerce catalog or directory in a modern browser, you scroll down and watch new items load cleanly into the grid. The interface feels responsive, but the underlying browser mechanism relies on JavaScript listening for scroll events or intersection observer triggers. Every time a user approaches the bottom of the page, the client script sends an asynchronous request to an API endpoint and appends new DOM nodes.

When an AI engine processes a prompt about your product category, it does not launch a full browser session to scroll down your page. An answer crawler such as ChatGPT-User, Perplexity-User, OAI-SearchBot, or Claude-SearchBot executes a direct HTTP GET request to your URL. It fetches the initial HTML response payload, extracts text content, and passes that text to the model to synthesize a response. It does not wait for asynchronous scripts to run, and it does not trigger scroll events.

If your product catalog relies entirely on infinite scroll script execution to reveal items beyond the first view, those deeper products are completely invisible to the answer engine. The crawler parses the initial document response, sees only the first few items rendered directly in the source markup, and evaluates your domain based on that limited view.

Answer crawlers versus training crawlers

To understand why infinite scroll breaks AI visibility, you must separate answer crawlers from training crawlers. Conflating these two infrastructure types leads to broken optimization strategies.

Training crawlers build the static datasets used to train future model weights. GPTBot, ClaudeBot, and Google-Extended are training crawlers. They operate asynchronously on massive web scales, collecting text that may influence model output months or years later.

Answer crawlers operate in real time to answer a user's prompt today. ChatGPT-User, Perplexity-User, PerplexityBot, OAI-SearchBot, Claude-SearchBot, and Claude-User are answer crawlers. When a user asks an engine for a product recommendation, the engine delegates retrieval to an answer crawler.

Because answer crawlers operate under strict real-time latency budgets, they cannot afford the computational overhead of running full headless browser environments for every page they visit. Executing JavaScript, waiting for network idle states, and simulating user scroll events takes seconds per page. Answer crawlers bypass this overhead by reading raw HTML payloads directly. If an item is missing from the initial server payload, the answer crawler will not observe it.

Blocking these bots in your configuration creates an immediate availability gap. In our crawler panel of 54,082 domains measured on August 8, 2026, 33,670 domains returned a readable robots.txt file. Out of those 33,670 readable domains, 5,497 block at least one AI crawler, and 2,576 block at least one answer crawler.

Looking at specific answer agents out of those 33,670 readable domains:

• PerplexityBot is blocked by 2,147 domains
• ChatGPT-User is blocked by 2,074 domains
• OAI-SearchBot is blocked by 1,579 domains
• Perplexity-User is blocked by 1,402 domains
• Claude-SearchBot is blocked by 1,400 domains
• Claude-User is blocked by 1,384 domains

For context, Googlebot is blocked by 610 of those same 33,670 domains. Training crawlers see higher block rates: GPTBot is blocked by 5,080 domains, ClaudeBot by 4,603 domains, and Google-Extended by 4,275 domains.

The block rates extend to early-stage technology companies as well. In our census of 4,226 active Y Combinator companies conducted on August 7, 2026, 3,755 returned a readable robots.txt file, and 253 of those domains blocked at least one AI crawler.

If your robots.txt allows answer crawlers access, but your application architecture hides catalog depth behind client-side scroll scripts, the net result for answer retrieval is identical to a block. The crawler requests the page, receives empty container markup, and leaves without your product data.

Diagnosing text visibility without JavaScript

You can test what an answer crawler sees on your catalog pages without buying specialized software. You need a command-line terminal to make raw HTTP requests that bypass browser script execution.

Open your terminal and run a simple curl command targeted at a specific product page URL deep in your infinite scroll feed:

curl -sL https://yourdomain.com/catalog | grep "Name of Deep Catalog Item"

If this command returns no text, the item does not exist in your initial HTML payload. The browser only renders the item because client-side JavaScript runs after document load.

You can also view the full text payload using a text-only terminal browser such as lynx:

lynx -dump https://yourdomain.com/catalog

Scroll through the output produced by lynx. What appears in this text stream is roughly what an answer crawler like ChatGPT-User or Claude-SearchBot processes during a real-time web search. If your page cuts off after ten products and shows a loading placeholder, the answer crawler receives that exact cutoff.

Search engines like Google spend significant infrastructure resources attempting to render client-side JavaScript via delayed rendering queues. Even so, Googlebot frequently misses content trapped behind dynamic scroll events that require explicit user interaction. Answer crawlers do not even attempt delayed rendering queues. They process the response payload immediately to feed text back to the language model.

The engineering realities of standard pagination

The fix for infinite scroll opacity is straightforward in theory: serve standard HTML pagination links using proper anchor elements.

Instead of relying solely on script events, every paginated state should expose explicit link markup:

<a href="/catalog?page=2">Next Page</a>

You can maintain an infinite scroll user experience for human site visitors by using progressive enhancement. When a human loads the page, client JavaScript intercepts the pagination link clicks and handles infinite scrolling smoothly. If a client does not execute JavaScript, such as an answer crawler, the underlying HTML links remain intact for traversal.

However, implementing standard pagination alongside client-side scroll script management involves real layout and design trade-offs:

• Layout shifts must be managed when server-rendering fallback catalog pages versus rendering client-side DOM insertions.
• URL state must sync cleanly across browser history, canonical tags, and server routes.
• Desktop and mobile viewport designs often require different visual pagination controls, increasing frontend maintenance effort.

Conceding these technical trade-offs is necessary, but you must also understand the limits of the fix. Implementing standard HTML links does not guarantee that an answer crawler will follow those links to page two, page three, or page ten.

Answer crawlers operate under restrictive crawl budgets per query. In many cases, an answer crawler fetches only the primary URL retrieved from a search engine result page. It may not follow secondary pagination links at all during a live query execution. Restructuring your catalog to include standard links ensures your deeper content is accessible, but it offers no promise regarding how deep a specific crawler will traverse on any given query.

What not to buy and how to measure real impact

When founders notice their catalog is missing from AI answer engines, vendors often try to sell quick fixes that do not address the underlying delivery architecture.

Do not buy agency packages or plugins that promise to fix AI indexing by adding schema markup or generating an llms.txt file. Schema markup embedded in the head of a page does not force a crawler to load unrendered dynamic products hidden behind client-side scroll events. Similarly, an llms.txt file is merely a markdown document pointing to raw text files. It does not alter how answer crawlers process your primary catalog pages. Standing has published evidence showing that schema markup and llms.txt files do not resolve underlying crawl access issues.

Do not buy services that offer guaranteed citations, guaranteed brand placements, or guaranteed rankings in AI engines. AI answer engines are non-deterministic systems. Asking ChatGPT the exact same question five times back-to-back will yield variations in source citations, brand selections, and text phrasing. Any vendor claiming a guaranteed ranking is selling an impossible promise.

Because AI answer engines are non-deterministic, measuring your brand's presence requires a statistical approach rather than a single check. A single test that returns your brand name is not a measurement; it is a single sample point.

At Standing, we measure brand presence by running a fixed basket of prompts five times per question across five major engines: ChatGPT, Perplexity, Gemini, Claude, and Google AI Overviews. Because answers fluctuate, we report citation scores with a confidence interval. A citation result of 34, plus or minus 6, at a 95% confidence interval represents a valid statistical band. A flat score of 34 without an interval is incomplete.

Most alternative tools run each prompt once, producing single-point scores that ignore model variance. They hide the non-deterministic nature of the engines they claim to measure.

If you are evaluating your brand's visibility across AI search engines, Standing provides transparent tracking across all five major engines:

• Track tier: $100 per month for 3 domains.
• Optimize tier: $300 per month for 5 domains with custom prompt basket configuration.
• Agency tier: $500 per month for 50 domains with a monthly re-scan.

Before purchasing any monitoring software or altering your marketing strategy, test your catalog pages in your terminal. If your deep product inventory relies on client-side scroll events that fail to render in raw HTML payloads, fix the underlying pagination markup first.