What is Googlebot?
Googlebot is the crawler that builds Google's core search index, and it is a different user agent from Google-Extended, which only controls whether Google's AI models can train on a page, so blocking one does not touch the other.
Googlebot is the original crawler, running since the 1990s, that fetches pages so they can rank in ordinary Google Search results. Google-Extended is newer and narrower: it only governs training use for Gemini and related models, and disallowing it leaves Googlebot's indexing untouched. The two are easy to conflate because both come from Google, but they answer completely different questions in robots.txt.
Blocking Googlebot is a far bigger decision than blocking Google-Extended, because it removes a site from Google Search entirely, and AI Overviews are generated from Google's regular search index rather than a separate crawl, so a Googlebot block removes a page from AI Overviews as a side effect. In a crawl of 33,670 sites with a readable robots.txt, only 610 (1.8%) disallowed Googlebot at all, far lower than the rate for any AI-specific crawler, consistent with how costly that tradeoff is.
Related
- Google-ExtendedGoogle-Extended is a robots.txt token that controls whether your content trains Gemini and grounds its answers. It is not a crawler and blocking it does not affect Google Search ranking.
- AI OverviewsAI Overviews are Google's AI-generated summaries shown above the traditional results for some searches.
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
- Training vs search crawlersA training crawler collects data models learn from. A search crawler builds the index an assistant cites when answering. Blocking one has nothing to do with the other.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.