robots.txt and sitemaps, done right

Three mechanisms decide what search engines crawl and show, and they're routinely confused. robots.txt controls whether a crawler fetches a URL. A sitemap tells crawlers which URLs exist and matter. The robots meta tag (noindex) controls whether a fetched page appears in results. Using the wrong one hides pages you wanted found — or exposes ones you didn't.

robots.txt

A single file at the domain root lists what crawlers may or may not fetch, and points to your sitemap. Keep it permissive for public content and specific about what to exclude (app internals, APIs).

User-agent: *
Disallow: /app
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

The sitemap

sitemap.xml lists your canonical, indexable URLs — the 200-returning ones you want found. Don't list redirects, noindex pages, or non-canonical variants. Reference it from robots.txt and submit it in Google Search Console.

<url>
  <loc>https://example.com/learn</loc>
  <changefreq>weekly</changefreq>
  <priority>0.7</priority>
</url>

noindex, for pages that should be crawled but not listed

<meta name="robots" content="noindex, follow">

Use this on thin, duplicate, or private-but-linked pages (login, account screens). "follow" still lets link equity flow through.

How to fix it

1. Serve a robots.txt at the root that disallows only what should never be crawled and links the sitemap.
2. List only canonical, 200 URLs in the sitemap; keep it in sync as pages change.
3. Use noindex (not Disallow) for pages you want crawled but not indexed.
4. Submit the sitemap in Search Console and re-check coverage.