Indexação · 5 min · 2026-05-31
Sitemap and robots.txt: what do they do and why are they different?
Sitemaps and robots.txt help crawlers in different ways. One supports discovery; the other controls access to selected paths.
Summary
What matters before moving forward.
- Sitemaps list relevant URLs for discovery.
- robots.txt controls crawling paths, not guaranteed indexation.
- Mistakes can hide pages or waste crawl resources.
A sitemap is a map of important URLs
An XML sitemap helps search engines find relevant pages, especially on new sites, catalogues, or large structures.
It should list final indexable pages, not redirects, errors, or duplicate URLs.
robots.txt defines crawling limits
robots.txt allows or blocks crawler access to paths. It is not the correct tool to remove an already indexed page.
An overly broad rule can accidentally block important resources or sections.
Related next steps
Where to keep exploring or apply this topic.
Proof and validation
- Validate implementation on the real deployment.
- Confirm technical signals against official documentation.
- Measure impact before promising outcomes.
Official documentation
References used to validate this approach.
Technical strategy should be auditable. These sources link directly to the official documentation behind this guide.
Frequently asked questions
Common questions about this topic.
Does a sitemap guarantee indexation?
No. It supports discovery, but every page still needs quality, coherent signals, and valid technical access.