How to Extract and Audit XML Sitemaps for SEO and Web Scraping

Overview

An XML sitemap is a machine-readable list of URLs that a site wants to make discoverable. It can contain individual page URLs, image or video references, or links to other sitemap files. A small site may publish one sitemap at a predictable path, while a larger site may use a sitemap index that points to multiple files grouped by content type, language, date, or product category.

A sitemap audit should not be treated as a direct substitute for crawling a website. A sitemap records what the site declares; an internal-link crawl records what can be discovered through links. Comparing the two views is often more informative than relying on either one alone. For a broader comparison, pair this workflow with an internal links audit.

Sitemap types to identify

For each URL, create a consistent record rather than storing only the raw text. A practical record might include:

• source_sitemap: the sitemap file where the URL was found;
• url: the normalized absolute URL;
• lastmod: the declared last-modified value, if present;
• http_status: the response status observed during validation;
• content_type: the returned media type;
• canonical_url: the canonical target when it is checked;
• redirect_target: the final or immediate destination, depending on your policy;
• internal_link_count: the number of internal links found in a separate crawl; and
• audit_notes: a controlled reason such as duplicate, blocked, redirected, or unavailable.

Checklist by scenario

1. Finding the sitemap

1. Check the site’s robots.txt file for sitemap declarations.
2. Inspect the site’s documented technical information or deployment configuration when you have access.
3. Test the expected sitemap location, but do not assume that one filename or path applies to every site.
4. Record the final response URL, status, content type, and any redirects.
5. Keep the original sitemap response as an audit artifact so later comparisons use the same input.

A sitemap URL is an input, not proof that the file is complete or valid. Confirm that the response contains XML rather than an HTML error page, access challenge, login form, or application-generated message.

2. Parsing a sitemap index

Read the root element before extracting URLs. A sitemap index points to child sitemaps, while a URL set contains the URLs you ultimately want to audit. Follow child references recursively, but maintain a visited-file set to prevent loops and repeated work.

Resolve every child reference as an absolute URL. If a file uses a relative reference, resolve it against the sitemap document’s own URL, not against the website homepage. Preserve the parent-child relationship in your output so a failed or stale child file can be traced back to the index.

3. Extracting and normalizing URLs

Extract the value of each URL element and retain relevant metadata such as lastmod. Normalize carefully:

• Resolve relative URLs against the source document.
• Apply a consistent hostname and scheme policy for comparison.
• Remove accidental surrounding whitespace.
• Decide whether URL fragments should be removed for the audit.
• Do not discard query parameters automatically; they may identify distinct resources, tracking variants, or unwanted duplicates.
• Store both the raw value and normalized value when reproducibility matters.

For large files, use a streaming XML parser rather than loading the entire document into memory. A lightweight Python workflow can look like this:

import gzip
import io
import requests
import xml.etree.ElementTree as ET
from urllib.parse import urljoin

response = requests.get(sitemap_url, timeout=30)
response.raise_for_status()
data = response.content

if sitemap_url.endswith(".gz"):
    data = gzip.decompress(data)

root = ET.parse(io.BytesIO(data)).getroot()
namespace = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

for node in root.findall("sm:url", namespace):
    location = node.findtext("sm:loc", namespaces=namespace)
    lastmod = node.findtext("sm:lastmod", namespaces=namespace)
    if location:
        print({"url": urljoin(sitemap_url, location.strip()), "lastmod": lastmod})

The exact parser and namespace handling may need adjustment for a particular file. Treat malformed XML, unexpected namespaces, and non-XML responses as explicit audit results rather than silently skipping them.

4. Validating crawlability and indexation signals

1. Request each URL with an appropriate rate limit and a clearly identified crawler where required by your operating environment.
2. Record status codes, redirects, timeouts, connection failures, and content types separately.
3. Flag URLs that return errors, redirect chains, non-HTML content, or an unexpected host.
4. Where the audit scope allows, compare the declared URL with its canonical target and relevant robots directives.
5. Compare sitemap URLs with internal crawl data and search performance or coverage data available to the site owner.

A successful HTTP response does not by itself establish that a URL is indexable, canonical, useful, or internally discoverable. Keep those dimensions separate in the report.

5. Exporting results

Export one row per normalized URL and a separate file for file-level errors. Useful columns include source_sitemap, url, lastmod, http_status, redirect_target, canonical_url, internal_link_count, and issue_type. If the output feeds another system, use stable field names and a consistent date format.

For additional data-quality practices, see this guide to cleaning, validating, and storing extracted web data.

What to double-check

XML and file handling

• Is the response actually XML, and does it have a supported root element?
• Does the document declare a namespace that your parser recognizes?
• Are compressed files decompressed before XML parsing?
• Are character encoding errors, truncated downloads, and malformed entities recorded?
• Does the sitemap index point to files that are reachable and parseable?

URL quality

• Are all extracted URLs absolute and associated with the expected site?
• Are duplicate URLs caused by case, trailing slashes, encoding, parameters, or alternate hostnames?
• Do URLs return the expected status and content type?
• Do redirects resolve to the preferred destination, or do they reveal migration leftovers?
• Are there URLs listed in the sitemap that have no internal links?

An orphan sitemap entry is not automatically a mistake. It may represent a newly published page, a landing page intentionally excluded from navigation, or a page whose internal linking needs review. Label it for investigation rather than applying an automatic fix.

Date comparisons

Compare lastmod values with observed page changes, deployment records, or crawl timestamps where possible. A changed date can indicate a meaningful update, a template regeneration, or an automated publishing process. It should be treated as a signal for review, not as proof that the page content changed.

Common mistakes

• Parsing only the first file: A sitemap index may contain the majority of the site’s URLs. Follow all valid child references.
• Assuming the URL path is the hostname: Resolve relative references against the file that contains them.
• Ignoring compression: A compressed response must be decompressed before it is passed to an XML parser.
• Using regular expressions as the XML parser: XML namespaces, escaping, and malformed input make a real XML parser safer and easier to audit.
• Counting every URL as unique: Normalize comparison keys while retaining the original URL for traceability.
• Equating a 200 response with indexability: Check redirects, canonical signals, robots directives, content quality, and internal discovery separately.
• Auditing too aggressively: Use controlled concurrency, caching, retries, and rate limits so validation does not create unnecessary server load.
• Overwriting historical results: Save dated exports and the input sitemap version so changes can be measured over time.

If the audit includes page-level extraction, keep sitemap discovery separate from browser automation. Many sitemap files can be handled with ordinary HTTP and XML tools; a browser is generally needed only for content that requires client-side rendering. See the web scraping tools comparison when choosing an extraction method.

When to revisit

Run this checklist before seasonal planning cycles, after a migration or domain change, when launching a large content group, and whenever a sitemap index or coverage report changes unexpectedly. It is also useful after changing a CMS, publishing workflow, URL structure, redirect rules, or XML generation process.

For recurring audits, compare the latest export with the previous one. Look for new error clusters, large changes in URL count, unexpected hostname changes, stale modification dates, newly duplicated paths, and URLs that moved from linked to orphaned status. Set thresholds appropriate to the site rather than relying on a universal number.

Refresh the workflow when your parser, crawler, storage system, or validation rules change. Recheck the sitemap protocol details and search-engine documentation periodically, especially before automating decisions based on the results. The practical final step is to turn findings into a prioritized action list: repair broken sitemap files first, remove or correct invalid entries, investigate redirects and duplicates, compare orphan URLs with business intent, and schedule a follow-up crawl to confirm the changes.