XML Sitemaps: What Belongs in One and What Does Not

Many sitemap files are treated as URL inventories, then filled with redirects, noindex pages, parameter variants, and every paginated URL the CMS can produce. That turns a useful discovery signal into a list search engines must second-guess: a sitemap is a recommendation, not a command.

The practical rule is simple: include URLs that return a successful 200 response, are indexable, and use a self-referencing canonical URL. Everything else needs a specific reason to be included, and most exceptions belong outside the sitemap.

What an XML sitemap is—and what it is not

An XML sitemap is a machine-readable list of URLs that a site wants search engines to discover and potentially consider for indexing. It can also provide optional metadata such as the last meaningful modification date. It does not force crawling, indexing, canonical selection, or ranking.

That distinction matters because many teams use a sitemap to compensate for poor internal linking or a large technical backlog. A sitemap can help search engines find important URLs, especially on a new, frequently updated, or very large site. It cannot make an unsuitable URL suitable for indexing.

Think of the file as a curated recommendation. A URL in it should meet three tests:

• It responds with 200 OK to a normal crawler request.
• It is eligible for indexing and is not blocked by a noindex directive.
• Its canonical tag points to itself, or the page has no conflicting canonical signal.

There is no general ranking penalty simply because a sitemap contains some unwanted URLs. The problem is operational and diagnostic: if 30% of the file consists of redirects, duplicates, or noindex pages, the file becomes less reliable as a statement of what the site considers important. It also makes reported sitemap coverage harder to interpret.

The inclusion rule: only recommend your preferred, indexable URLs

A sitemap should reflect the final URL set, not the journey a user or crawler takes to reach it. If an old product URL redirects to a replacement, the replacement belongs in the sitemap after it returns 200 OK; the old URL does not.

The same principle applies to canonicalisation. If five URLs display substantially the same content but four canonicalise to one preferred URL, put the preferred URL in the sitemap. Do not list all five and expect the canonical tag to resolve the contradiction cleanly.

Indexability must be checked in context. A page can return 200 OK and still be unsuitable because it has a noindex meta tag, an X-Robots-Tag: noindex header, thin or incomplete content, or a canonical pointing elsewhere. A successful HTTP response is necessary, not sufficient.

Use lastmod only for meaningful changes to the page’s primary content, structured data, or important navigational elements. Changing the timestamp whenever a job runs is not an accurate update signal. A daily timestamp on hundreds of thousands of unchanged URLs tells search engines little about what actually changed.

URL-by-URL decision table

URL type	Should it be in the sitemap?	Why
Indexable page returning 200 OK with a self-canonical	Yes	This is the core set of URLs the site wants discovered and considered for indexing.
Redirecting URL, such as 301 or 302	No	Recommend the final destination instead. A redirect is a route to a page, not the preferred page itself.
URL with a noindex directive	No	It explicitly asks search engines not to index the URL, so including it creates a direct contradiction.
Canonicalised duplicate	No	List the canonical destination, not an alternate URL that declares another page as preferred.
Blocked URL in robots.txt	Usually no	Search engines may be unable to fetch the page and verify its content or indexability. Fix the access decision before recommending the URL.
Faceted navigation or parameter URL	Usually no	These often create near-duplicate combinations, unstable URL sets, or pages with little standalone search value.
Search results page	No	Internal search pages usually should not be part of the site’s indexable content set.
Paginated listing page, such as page 2 or page 3	Usually no	For most sites, the valuable URLs are the category or collection page and the individual detail pages, not every pagination state.
Image, video, or news sitemap entry	Sometimes	Use the relevant specialised sitemap when the media or publication content is genuinely important and meets its own eligibility requirements.

The word “usually” matters. Pagination is not automatically forbidden, and parameters are not automatically worthless. If a paginated URL has unique, indexable content, a stable canonical, search demand, and a deliberate internal-linking role, it may qualify. That should be a documented editorial decision, not an automatic export from a CMS.

What not to put in a sitemap

Noindex pages

Do not include login pages, internal search results, thin utility pages, expired campaigns, thank-you pages, or any other URL carrying noindex. The popular advice that “including noindex URLs helps Google discover them before removing them” is wrong for a normal production sitemap. The sitemap should communicate the pages you want indexed, not pages you have deliberately excluded.

If a page is temporarily noindex during a launch, remove it from the sitemap until the directive is removed and validation confirms the page is indexable. If the page is permanently excluded, keep it out permanently.

Redirects and chains

A sitemap containing redirected URLs creates avoidable work. It can also hide redirect chains, such as an old URL going to an intermediate URL before reaching the final page. Replace sitemap entries with the final destination and investigate any chain longer than one redirect.

Do not use a sitemap as a historical archive of old URLs. Redirect maps belong in deployment documentation, analytics annotations, or migration testing—not in the live recommendation file.

Canonical conflicts

If a page has a canonical pointing to another URL, leaving it in the sitemap tells search engines two different things: “consider this URL” and “prefer that other URL.” Fix the canonical relationship first. In most cases, the canonical URL—not the duplicate—should be listed.

Parameter and faceted URLs

Filters such as ?colour=blue, sorting parameters, tracking parameters, session identifiers, and combinations of multiple facets can generate thousands of URL variants. Adding them all to a sitemap does not make them valuable landing pages. It often exposes duplication, crawl traps, and inconsistent canonical signals.

Include a parameter URL only when it represents a deliberately indexable page with unique content, a stable URL, a self-canonical, and a clear business or search purpose. A filtered category page created solely because a customer selected an option normally fails that test.

Every pagination state by default

Another popular recommendation is to place every paginated URL in the sitemap “so crawlers can reach all products.” That is often wrong. Good internal links from the category page, pagination controls, XML sitemap entries for product pages, and appropriate crawl paths may already provide discovery. Adding page 2 through page 400 can overwhelm the file with low-value listing states.

There are exceptions for editorial archives, forums, or genuinely unique paginated collections. Assess the content and canonical rules rather than applying a universal pagination setting.

How to build a trustworthy sitemap generator workflow

A sitemap generator should be judged by the quality of its filtering and validation, not just by whether it produces valid XML. Whether the tool is a crawler, CMS module, or custom script, the workflow should perform these checks before publishing:

1. Collect canonical URL candidates from the site’s approved content sources.
2. Resolve relative URLs and normalise protocol, host, trailing-slash, and case conventions.
3. Request each candidate or consume reliable crawl data to confirm the final HTTP status.
4. Exclude redirects, error responses, blocked pages, noindex URLs, and canonicalised duplicates.
5. Compare the final URL against its canonical tag and exclude conflicts.
6. Remove tracking parameters, session identifiers, and unapproved faceted combinations.
7. Set lastmod from a real content-change field, not the generation timestamp.
8. Validate XML syntax, encoding, URL escaping, file size, and sitemap index references.
9. Publish only after a sample review and a comparison with the previous version.

Set operational thresholds before the project starts. The XML Sitemap protocol allows up to 50,000 URLs or 50 MB of uncompressed data per sitemap file, according to the official sitemaps.org protocol documentation. Split earlier if generation, validation, or review becomes slow; a practical internal limit of 10,000 to 25,000 URLs per file can make troubleshooting easier on large sites, but it is an operating choice rather than a search-engine requirement.

Run a full validation after major migrations, template changes, or canonical updates. For a stable site, a daily or weekly generation schedule can be reasonable, but validation should happen every time the file changes. A monthly manual review is a useful minimum for smaller sites; high-change ecommerce sites may need checks every 24 hours.

How to measure sitemap quality

Do not assess success by the number of URLs in the file. Assess whether the file accurately represents the site’s indexable inventory.

• Validity rate: target 100% of entries returning 200 OK in a controlled validation sample or full crawl.
• Indexability alignment: target 100% of entries without a noindex directive or canonical conflict.
• Redirect contamination: target 0% of entries resolving through a redirect.
• Freshness accuracy: investigate any large group with identical timestamps, especially if no corresponding content release occurred.
• Coverage gap: compare important indexable URLs discovered through internal links, databases, and logs against sitemap membership.

Use samples when the site is too large for a full check on every deployment. For example, review 200 URLs from each major template, plus every URL added or changed during the release. A 200-URL sample is not proof that the whole file is clean, so perform a complete crawl after migrations and high-risk technical changes.

Search Console sitemap reports can help identify submitted-versus-indexed differences, but a gap is not automatically a sitemap failure. Some valid URLs are not indexed because of quality, duplication, demand, or search-engine decisions. Investigate patterns by template instead of treating every unindexed URL as a technical error.

Sitemap architecture for large and specialised sites

Use separate sitemap files when segmentation makes ownership and diagnosis easier. Common groups include products, categories, editorial content, images, videos, or news content. Keep each group internally consistent so a spike in errors can be traced to one template or publishing system.

An index sitemap can reference multiple child sitemaps. Keep the index current: remove deleted child files, update references when filenames change, and ensure every child file is accessible from the submitted index. A sitemap index is an organisational layer, not permission to include lower-quality URLs.

For international sites, include the correct canonical URL for each language or regional version and keep international targeting annotations consistent with the URL set. Do not use a sitemap to conceal inconsistent hreflang, duplicate regional pages, or accidental redirects between markets.

For media, use specialised sitemap information only when it describes content that is actually present and valuable on the page. A video entry for a removed player, or an image entry pointing to a blocked asset, adds noise rather than improving discovery.

A practical governance standard

Assign one owner for sitemap logic and require changes to pass three checks: URL eligibility, canonical consistency, and deployment validation. Store the generation rules in version control where possible, including exclusions for parameters, templates, environments, and content states.

Before a release, ask four direct questions:

• Would the business want this exact URL indexed?
• Does it return 200 OK without a redirect?
• Does it have a self-canonical and no noindex directive?
• Would a human reviewer understand why it belongs in the recommended URL set?

If the answer to any question is no, exclude the URL or fix the underlying site signal first. The strongest XML sitemap is not the largest one. It is the one whose entries consistently match the site’s stated indexing preferences, making the file a precise recommendation rather than an automated dump of everything the CMS happens to know about.

Related reading

• Content strategy — scope, method and what we measure.
• AI search optimisation
• Why Every Backlink Tool Shows a Different Number
• Website Change Monitoring: Catching Silent SEO Regressions