What is sitemap.xml?
sitemap.xml is a file that lists a site's URLs for crawlers to discover, and it's a discovery aid, not a permission system, so listing a page in a sitemap does not override a Disallow rule that blocks a crawler from fetching it.
A sitemap is essentially an index of URLs a site wants crawled, often with metadata like when a page last changed. It's commonly referenced from within robots.txt using a Sitemap line, which is one reason the two files get bundled together in people's minds, but they do different jobs: robots.txt sets permissions, sitemap.xml just points to what exists.
Because a sitemap can't grant access, adding a page to it doesn't help if that page's path is also disallowed elsewhere in robots.txt, the crawler will see the URL listed but still won't fetch it. Sitemaps matter most for large or poorly linked sites where a crawler might otherwise miss pages through normal link-following; for a small, well-linked site, skipping a sitemap rarely changes what gets crawled.
Related
- robots.txtrobots.txt is a file at a site's root that tells crawlers which parts of the site they may fetch. It is a request, not an enforcement mechanism.
- Crawl budgetCrawl budget is how much crawling a search engine is willing to spend on your site. On a new or low-authority domain it is small, which is why publishing more pages can reduce how many get indexed.
- IndexationIndexation is whether a search engine has stored your page and is willing to return it. Being crawled is not the same as being indexed, and being indexed is not the same as ranking.
Want to know where you actually stand on this? Run a free visibility check or try the free tools.