A reliable technical SEO audit starts with a consistent page-level record. This checklist explains how to extract HTTP status codes, title tags, meta descriptions, canonical URLs, robots directives, headings, and internal links, then turn those fields into actionable groups such as missing, duplicate, redirected, or conflicting values.
Overview
Technical SEO data extraction is the process of requesting a set of URLs, reading the response and rendered HTML where necessary, and storing comparable fields for review. The goal is not to collect every possible element on a page. It is to capture the signals that help you understand whether important URLs are accessible, indexable, descriptive, internally connected, and consistent with the site’s intended architecture.
A useful URL record should preserve both the requested URL and the final result. Redirects, protocol changes, trailing-slash rules, and host variations can otherwise disappear during normalization. At minimum, create one record per requested URL with these fields:
- Requested URL: the URL discovered in a sitemap, link, export, or audit list.
- Final URL: the last URL after redirects, if different.
- HTTP status: the response status observed by the crawler.
- Redirect chain: each hop, status, and destination when applicable.
- Content type and retrieval result: useful for separating HTML pages from files, APIs, errors, or blocked responses.
- Title and meta description: store the text and character or pixel-length measurement used by your workflow.
- Canonical URL: the normalized value found in the page markup, if present.
- Robots directives: page-level directives from the HTML and relevant response headers.
- Heading structure: counts and text for H1 through H6, with particular attention to the primary heading.
- Internal links: absolute or normalized destinations, anchor text, and optionally link attributes.
Keep raw values alongside cleaned values. For example, retain the original title text but also store a trimmed version for duplicate detection. This makes the audit reproducible and helps explain why two records were grouped together.
For context, combine page extraction with XML sitemap data and internal-link data rather than treating the page list as complete. The guide to extracting and auditing XML sitemaps covers a useful source of URLs, while the workflow for auditing internal links at scale helps reveal pages that are linked, orphaned, or linked from unexpected areas.
Checklist by scenario
For a new crawl or baseline audit
- Define the URL sources: XML sitemaps, navigation, internal links, analytics exports, server logs, or a supplied list.
- Normalize obvious duplicates without erasing the original requested URL. Decide how to handle fragments, parameter order, case, hostnames, and trailing slashes.
- Request each URL with a rate limit and a clear user agent. Respect applicable access rules and avoid creating unnecessary load.
- Record the full response path, including status codes, redirects, content type, and retrieval errors.
- Parse HTML only when the response is suitable for HTML parsing. Do not treat an error page, JSON response, or PDF as a normal document.
- Extract the title, meta description, canonical link element, robots meta directives, headings, and internal links.
- Group records by status, final URL, canonical URL, title, description, and normalized heading text.
For a migration or URL-structure change
- Compare the old URL set with the new URL set before launch and after redirects are active.
- Check whether old URLs resolve to the intended new equivalents rather than a generic destination.
- Identify redirect chains, loops, unexpected status codes, and important URLs that return an error.
- Compare canonical values, internal links, sitemap entries, and hreflang or structured data references if those are part of the project.
- Re-crawl a representative sample of templates as well as high-value URLs. A single successful page does not confirm that every template behaves correctly.
For a content or template audit
- Find missing, empty, and duplicate title tags and meta descriptions.
- Review unusually short or long values as review queues, not automatic proof of an error.
- Compare page type with heading patterns. Product, category, article, and utility templates may require different expected fields.
- Check whether canonical URLs are absent, malformed, non-absolute, redirected, or inconsistent with the final URL.
- Inspect pages with no internal links or with links pointing to obsolete, redirected, or non-canonical destinations.
For recurring technical SEO automation
Store each crawl with a timestamp and compare it with the previous run. Prioritize changes rather than sending a complete list every time. Useful alerts include a sudden increase in non-success responses, new duplicate titles, canonical changes on stable templates, newly missing descriptions, or a drop in internal-link coverage. A monitoring process should also track crawler failures so a network problem is not mistaken for a site-wide SEO issue. See the guide to web scraper monitoring and data-quality checks for a broader validation approach.
What to double-check
HTTP status codes: Separate the requested response from the final response. A URL that redirects and a URL that directly returns a successful response are not the same audit result. Also record timeout, DNS, TLS, and parsing failures separately from HTTP errors.
Canonical URLs: Extract the href value, resolve relative references, and normalize it using the same rules applied to other URLs. Then compare it with the final URL and the site’s intended preferred version. A canonical is a signal to review, not a replacement for checking status, accessibility, and internal linking.
Robots directives: Capture directives from both HTML metadata and applicable response headers. Store the raw directive string as well as parsed flags such as noindex or nofollow. Do not confuse page-level directives with a site’s robots.txt rules; they are different inputs and should be audited separately.
Titles and descriptions: Preserve visible text exactly before whitespace cleanup. Detect empty values, duplicates, and template boilerplate, but send borderline lengths to editorial review. Length thresholds can vary by display context and should not be treated as universal pass-or-fail rules.
Headings: Record heading order and text, not just the number of H1 elements. A heading count can identify patterns, but it cannot determine whether the page’s information hierarchy is clear without context.
Internal links: Resolve relative links, remove fragments for destination grouping, and retain anchor text separately. Exclude or classify navigation, footer, pagination, and utility links when interpreting page-level link coverage. For selector design that survives template changes, review this CSS and XPath selector guide.
Common mistakes
- Crawling only the sitemap: Sitemaps can omit linked pages, contain stale URLs, or include URLs that are not internally discoverable. Use multiple sources where appropriate.
- Keeping only the final URL: This hides redirect behavior and makes migration defects harder to diagnose.
- Using one parser for every response: Check content type and status before parsing. HTML, JSON, PDFs, and error documents need different handling.
- Relying on browser rendering for every page: Start with direct HTTP extraction when the required HTML is present. Use a browser only when client-side rendering is necessary, and record which method produced each result.
- Overwriting raw data during cleaning: Keep raw HTML fields or response metadata where permitted, then create derived columns for normalized comparisons.
- Calling every duplicate an error: Similar titles or descriptions may be intentional on some page types. Group first, then review according to template and search intent.
- Ignoring crawl scope: Parameters, calendars, filters, session URLs, and infinite scroll can create large volumes of low-value requests. Define exclusions and maximum crawl limits before starting.
- Combining extraction with unsupported conclusions: A missing tag or status pattern identifies a review opportunity. It does not, by itself, explain rankings, traffic, or indexing outcomes.
When the output will feed another system, validate the schema before loading it. A practical pipeline should clean, normalize, validate, and store extracted records consistently; the web scraping data pipeline design guide provides a useful framework for that step.
When to revisit
Run this checklist before seasonal planning cycles, after a significant template release, during a migration, and whenever redirect or canonical rules change. It is also worth scheduling recurring checks for sites where content and URLs change frequently. The right interval depends on crawl volume, release frequency, and the cost of an undetected error; consistency matters more than choosing a universal schedule.
Refresh the workflow when the crawler, parser, rendering method, storage schema, or URL-normalization rules change. Recheck extraction logic after a redesign because selectors, heading structures, navigation elements, and metadata placement may move. Review robots and canonical guidance against current search-engine documentation when policies or implementation recommendations change rather than copying assumptions from an older audit.
Before closing an audit, save the crawl date, scope, request settings, exclusions, parser version, and comparison rules. Then turn the findings into a short action list:
- Fix or investigate access and redirect anomalies first.
- Review canonical and robots conflicts on important URLs.
- Group metadata and heading issues by template so one fix can address many pages.
- Validate internal-link changes with a focused recrawl.
- Store the final dataset and unresolved exceptions as the baseline for the next check.
This approach makes technical SEO data extraction a repeatable diagnostic workflow rather than a one-time spreadsheet exercise. The result is a clearer record of what changed, where review is needed, and which fixes should be tested next.