What is crawl budget?
Crawl budget is how much crawling a search engine is willing to spend on your site. On a new or low-authority domain it is small, which is why publishing more pages can reduce how many get indexed.
It is allocated on something like trust. An established site with links pointing at it gets crawled deeply and often; a new domain gets sampled. This is the mechanism behind a counterintuitive outcome: shipping hundreds of pages at once on a young site can leave fewer of them indexed than shipping a handful, because the crawler spends its allowance discovering near-duplicates and concludes the site is thin.
The practical rule that follows is about ratio rather than count. A sitemap where every URL is distinct and substantial teaches a crawler the site is worth returning to. One that is mostly templated variations teaches the opposite, and that lesson is expensive to unteach.
Related
- IndexationIndexation is whether a search engine has stored your page and is willing to return it. Being crawled is not the same as being indexed, and being indexed is not the same as ranking.
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
- Thin contentThin content is a page that exists to be indexed rather than to be read. Google's test is purpose and added value, not word count.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.