How to Find Broken Links, Redirect Chains, and Dead Pages with a Website Crawler
broken-linksredirectstechnical-seowebsite-crawlinghttp-status-codessite-audits

How to Find Broken Links, Redirect Chains, and Dead Pages with a Website Crawler

WWeb Dev Toolbox Editorial Team
2026-11-17
7 min read

Learn how to crawl a website for broken links, redirect chains, 404 errors, orphaned pages, and high-priority internal-link fixes.

A reliable website crawl can reveal broken internal links, unnecessary redirect hops, inaccessible pages, and URLs that no longer belong in your site structure. This checklist explains how to configure a crawl, interpret HTTP responses, trace redirect chains, find orphaned pages, and prioritize fixes using evidence from internal links, sitemaps, and analytics exports.

Overview

A broken-link audit is more useful when it is treated as a data workflow rather than a simple list of 404 errors. A crawler should record the URL it discovered, the page that linked to it, the final response, the redirect path, and any relevant page-level signals. That context lets you distinguish a high-priority navigation problem from an obsolete URL that is no longer referenced.

Before starting, define the crawl boundary. For a normal internal-link audit, begin with the preferred HTTPS hostname and decide whether subdomains, staging paths, media files, JavaScript assets, PDFs, and query-parameter variations belong in scope. Respect the site’s access controls, use a reasonable request rate, and confirm that the crawl is authorized. The web scraping legal and ethical pre-crawl framework provides a useful preparation checklist.

Use a crawl configuration that can:

  • Start from one or more representative internal URLs.
  • Follow internal links while applying a clear hostname and path rule.
  • Record source URL, destination URL, anchor text, link type, status code, and final URL.
  • Follow redirects and preserve every hop in the redirect path.
  • Export results to CSV or a spreadsheet for filtering and prioritization.
  • Optionally compare discovered URLs with XML sitemaps, navigation exports, and analytics data.

HTTP-status reference

ResponseWhat it usually meansAudit action
200The server returned the requested resource.Check that the page is the intended destination and is not a soft error page.
3xxThe URL points elsewhere or requires another request.Trace the chain, confirm the destination, and replace unnecessary internal redirects.
4xxThe request was not fulfilled for a client-side reason, such as a missing or restricted resource.Repair, replace, or intentionally remove the internal reference.
5xxThe server failed to complete the request.Recheck the URL and timing, then investigate hosting, application, or capacity issues.

Status codes are evidence, not the complete diagnosis. A URL can return 200 while displaying an error message, while a deliberately removed page may have a valid business reason for returning a non-200 response. Review the page context before changing anything.

Checklist by scenario

  1. Choose crawl seeds. Include the homepage, primary navigation destinations, important category pages, and a representative selection of templates. Add XML sitemap URLs when available; the XML sitemap audit workflow explains how to inspect them as a separate URL source.
  2. Crawl HTML links. Capture both the linked URL and the source page. Store anchor text and whether the link appears in navigation, content, breadcrumbs, footer elements, or another repeated component.
  3. Filter response problems. Review 4xx and 5xx results first. Recheck suspicious failures because temporary server errors, rate limits, authentication requirements, or crawler settings can produce misleading results.
  4. Check the source context. A broken link in the main navigation or a popular article deserves earlier attention than an obsolete reference in a low-value archive.
  5. Choose a fix. Update the link to the correct live URL, restore the intended resource, remove the reference, or redirect the old URL when a relevant replacement exists.

Scenario 2: Trace redirect chains

  1. Export every redirecting URL and its complete hop sequence, not only the final destination.
  2. Separate direct redirects from chains. A chain exists when URL A sends the request to URL B, which sends it to URL C or beyond.
  3. Check for loops, alternating HTTP and HTTPS destinations, hostname changes, unnecessary trailing-slash changes, and redirects that end at an unrelated page.
  4. Update internal links so they point directly to the preferred final URL. Retain an old-URL redirect when it is needed for external references, bookmarks, or a migration plan.
  5. Test the final destination independently for status, canonical signals, page content, and accessibility to the crawler.

Redirect cleanup should be coordinated with the site owner or developer when server rules, application routing, or a migration is involved. Do not remove redirects simply because they appear old; first determine whether the old URL still has a legitimate use.

Scenario 3: Find dead and orphaned pages

A crawler discovers pages through links and other configured URL sources. A page can therefore exist in an XML sitemap or analytics export without being reachable through the current internal-link graph. To find these cases, compare three sets:

  • Crawled URLs: pages reached by following links from your seeds.
  • Sitemap URLs: URLs declared by the site’s XML sitemap files.
  • Known historical or active URLs: URLs from analytics, server logs, a CMS export, or a previous crawl, where those data sources are available.

URLs that appear in a sitemap or historical export but have no incoming internal link are candidates for an orphan-page review. Confirm that they are live, useful, and intended to remain part of the site. Add contextual links where appropriate, update the sitemap if the URL should not be included, or retire it through an intentional content decision.

Use one row per discovered link or one row per unique URL, depending on the audit’s purpose. A link-level sheet is better for fixing source pages; a URL-level sheet is better for consolidating response and redirect data. Useful columns include:

source_url, destination_url, anchor_text, link_location, status_code, final_url, redirect_chain, content_type, canonical_url, in_sitemap, incoming_link_count, template_or_section, priority, owner, recommended_action, and verification_date.

For larger audits, extract internal links separately and join them to crawl responses. The guide on auditing internal links at scale can help structure that dataset.

What to double-check

  • Canonical and destination alignment: A page may return 200 but canonicalize to another URL. Record both values and investigate deliberate versus accidental differences. See the guide to extracting status codes, canonicals, and meta tags.
  • Internal versus external links: Keep external failures in a separate queue. They may require different verification steps and ownership.
  • URL normalization: Decide how to compare case, fragments, trailing slashes, encoded characters, and query parameters. Apply the same rule throughout the export.
  • Rendering requirements: If important links appear only after JavaScript execution, compare an HTML crawl with a browser-rendered crawl. Browser automation can be useful, but configure it carefully and keep the crawl scope controlled.
  • Authentication and blocked areas: A private page may fail because the crawler lacks a session rather than because the URL is dead. Do not expose credentials in crawl exports. For background on session-aware extraction, see the guide to cookies, sessions, and CSRF tokens.
  • Selector quality: If links are extracted from changing layouts, use stable attributes and test selectors against several templates. The guide to resilient CSS and XPath selectors covers this problem.

Prioritize by impact

A practical priority model combines response severity with internal importance. Start with broken links from primary navigation, high-traffic pages, conversion paths, and pages with many incoming links. Next review redirect chains repeated across a template. Then handle isolated references, low-value archives, and URLs with no evidence of current use. Record the reason for each priority so the queue remains understandable after the audit.

Common mistakes

  • Calling every 404 an error: Some removed URLs are intentional. The key question is whether a live page still links to them or whether a relevant replacement should exist.
  • Checking only the homepage: Deep pages, filtered listings, help content, and legacy templates often contain different link patterns.
  • Reporting only final status: Without the source page and redirect path, developers cannot see what should be changed.
  • Ignoring repeated components: One obsolete link in a footer or navigation template can create hundreds of broken references.
  • Confusing orphaned with unimportant: An unlinked page may still be valuable, current, or referenced externally. Review its purpose before removing it.
  • Making fixes without a second crawl: Re-run the affected scope and verify that the source links, status codes, final destinations, and redirect paths now match the intended result.

When to revisit

Repeat the audit on a schedule appropriate to the site’s change rate, and always revisit it after a migration, redesign, navigation change, hosting change, large content launch, or redirect-rule update. It is also useful before seasonal planning cycles when important landing pages and campaign paths are being reviewed.

Keep the previous export, crawl configuration, exclusions, and date in a shared location. Comparing runs makes it easier to distinguish newly introduced failures from known exceptions. When workflows or crawler tools change, confirm that rendering, URL normalization, redirect handling, and rate limits still produce comparable data.

Use this final action list for the next audit:

  1. Define the hostname, paths, URL types, and access rules in scope.
  2. Collect crawl seeds and supporting URL sources such as sitemaps or CMS exports.
  3. Run the crawl and export source links, response codes, final URLs, and redirect paths.
  4. Recheck 4xx and 5xx results, then classify redirect chains and orphan candidates.
  5. Prioritize fixes by template reach, internal-link importance, and page purpose.
  6. Assign owners, record recommended actions, and preserve the audit baseline.
  7. After changes, crawl the affected areas again and close only verified issues.

Related Topics

#broken-links#redirects#technical-seo#website-crawling#http-status-codes#site-audits
W

Web Dev Toolbox Editorial Team

Technical SEO Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.