Sitemaps, Robots.txt, Canonical URLs and Indexing

A practical technical SEO guide explaining discovery versus indexing, robots.txt placement, canonical tags, and Search Console verification.

Discovery is not the same as indexing

A search engine first needs to discover a URL, then crawl it, evaluate its content and potentially index it. A URL appearing in a sitemap only assists discovery. It is not a command to include the page in search results, and neither an HTML canonical tag nor a request in Search Console guarantees indexing.

To investigate a missing page, begin with Google Search Console → URL Inspection, not by editing every SEO tag. Check whether the page is known to Google, whether crawling was allowed, the last fetch result, the selected canonical and any “Discovered – currently not indexed” or “Crawled – currently not indexed” status.

A sitemap is a truthful inventory

A sitemap should contain canonical, useful pages that actually exist on the website. Add the exact public URLs, including the correct path and trailing slash. Do not include missing pages, placeholder routes or tracking URL variants. See XML Sitemap Generator for creating XML from a controlled URL list.

For example, if your site lives at https://example.github.io/my-project/, a tool page might live at https://example.github.io/my-project/tools/example.html. Listing https://example.github.io/tools/example.html would incorrectly point to the root website, not the project.

Robots.txt is read at the host root

The standard crawler file belongs at https://host.example/robots.txt, not at an arbitrary project subdirectory. For a GitHub Pages project below a user site's hostname, the project directory's own robots.txt is not the controlling crawler file. The user site's root /robots.txt controls crawling for paths under that hostname.

Important: An Allow: / rule does not force indexing; it simply does not block crawling. Use a noindex directive on a page that should not be indexed, rather than using a robots block and expecting the page to disappear from search.

Canonicals should reflect the real preferred URL

If the homepage is accessible at both /my-project/ and /my-project/index.html, choose one canonical, usually the directory URL. Tool pages should normally point to their own real full public URL. Do not make all tool pages canonical to the homepage; that would tell search engines that the individual tools are not your preferred versions.

How to diagnose a real problem

  1. Open the final published page in an incognito browser and inspect whether the core content is visible.
  2. Use a browser developer console to check for missing CSS, JavaScript, fonts or data files.
  3. Confirm that noindex is absent on pages you intend to publish.
  4. Inspect the canonical and make sure the selected URL exists and is not a different tool.
  5. Compare the XML sitemap against actual published pages.
  6. Use Search Console's URL Inspection and Page indexing report to distinguish a technical block from a content-quality or discovery delay.

Helpful official references

Frequently asked questions

Does submitting a sitemap mean Google will index everything?

No. Google chooses which discovered pages to crawl and index.

Can a project-folder robots.txt change the root host crawler rules?

No. Crawler rules are read from the hostname's root robots.txt.

Should I submit hundreds of weak pages at once?

Prioritize useful, functioning pages with original explanations and reliable navigation. Extra URLs do not compensate for low-value or duplicate content.

← More practical guides