Indexação · 5 min · 2026-05-31

Sitemap and robots.txt: what do they do and why are they different?

Sitemaps and robots.txt help crawlers in different ways. One supports discovery; the other controls access to selected paths.

Sitemaprobots.txtCrawl

Summary

What matters before moving forward.

  • Sitemaps list relevant URLs for discovery.
  • robots.txt controls crawling paths, not guaranteed indexation.
  • Mistakes can hide pages or waste crawl resources.

A sitemap is a map of important URLs

An XML sitemap helps search engines find relevant pages, especially on new sites, catalogues, or large structures.

It should list final indexable pages, not redirects, errors, or duplicate URLs.

robots.txt defines crawling limits

robots.txt allows or blocks crawler access to paths. It is not the correct tool to remove an already indexed page.

An overly broad rule can accidentally block important resources or sections.

Related next steps

Where to keep exploring or apply this topic.

Proof and validation

  • Validate implementation on the real deployment.
  • Confirm technical signals against official documentation.
  • Measure impact before promising outcomes.

Official documentation

References used to validate this approach.

Technical strategy should be auditable. These sources link directly to the official documentation behind this guide.

Frequently asked questions

Common questions about this topic.

Does a sitemap guarantee indexation?

No. It supports discovery, but every page still needs quality, coherent signals, and valid technical access.

Request technical analysis Read in Portuguese

Want to apply this to your site?

Next step Technical audit