SEOPlaywrightAudit Tools

Automated SEO Audit Spider: Playwright + Lighthouse for JavaScript-Heavy Sites

UUnknown

2026-02-28

10 min read

Build a Playwright+Lighthouse spider that renders JS, extracts JSON-LD and entity signals, and generates actionable SEO audits.

Hook: Why JavaScript-heavy sites break ordinary SEO audits

If your SEO audit keeps missing issues on single-page apps, progressive web apps, or sites that hydrate content client-side, you know the pain: crawlers report green, but organic traffic stalls. The root cause is simple — most audits run on raw HTML and miss the rendered DOM, structured data injected by JavaScript, and the runtime signals search engines use today.

Executive summary — what you'll build and why it matters

This guide shows how to build a reusable automated SEO audit spider that renders pages with Playwright, captures full page metrics with Lighthouse, extracts structured data (JSON‑LD) and basic entity signals, and produces actionable audit reports you can feed into BI, ticketing, or pipelines. The patterns work at scale, handle JS-heavy pages, and are tuned for 2026 search realities like entity-first ranking signals and generative SERP features.

Why this approach in 2026?

Entity SEO and knowledge-graph signals are central to search features: audits must capture schema, sameAs/linked entities, and content that disambiguates entities.
Search result presentation evolved: generative and multi-modal SERPs rely on runtime signals (rendered content, structured data) more than static meta tags.
JavaScript usage is ubiquitous: rendering at crawl time is required to surface the content search engines evaluate.
Better toolchain integration: Playwright's CDP connectivity and Lighthouse's programmatic API let us pair rendering with production-grade audits.

What the spider captures (audit checklist)

Lighthouse metrics (Performance, Accessibility, Best Practices, SEO, PWA) as JSON + HTML
Structured data (all application/ld+json blocks, microdata items, and detected schema.org types)
Entity signals (main entities, sameAs links, prominent named entities from headings and intro paragraphs)
Technical signals (status code, canonical, hreflang, robots directives, server timing)
Runtime DOM observations (rendered H1, content fallback, lazy-loaded critical content presence)

High-level architecture

The spider is modular: a crawler/queue orchestrates URLs → worker(s) that drive a shared browser instance → collectors that extract data → an aggregator that merges Lighthouse output with extracted structured data and entity signals → exporters (JSON, CSV, HTML, or push to Elastic/BigQuery).

Why combine Playwright + Lighthouse?

Playwright renders the page and lets you run deterministic scripts to wait for hydration, network idle, or custom DOM signals.
Lighthouse produces standardized, comparable metrics and audits used by SEOs and engineers for prioritization.
By starting the browser once and connecting both Playwright and Lighthouse to the same remote debugging port, you get consistent snapshots and avoid race conditions between rendering and auditing.

Prerequisites

Node.js 18+ (2026 LTS recommended)
npm/yarn
Packages: playwright, lighthouse, chrome-launcher, p-queue (or similar), csv-writer
Optional: proxy provider, CI runner with enough CPU to run Chromium headless

Step-by-step build: core worker (Playwright + Lighthouse)

The example below uses chrome-launcher to start a Chromium instance on a debugging port, Playwright to render pages over CDP, and lighthouse programmatically to run audits against the same browser. This pattern gives single-run consistency between the rendered DOM and Lighthouse metrics.

Install dependencies

npm install playwright lighthouse chrome-launcher p-queue fs-extra csv-writer

Example worker (index.js)

const chromeLauncher = require('chrome-launcher');
const playwright = require('playwright');
const lighthouse = require('lighthouse');
const fs = require('fs-extra');

async function startChrome() {
  const chrome = await chromeLauncher.launch({
    chromeFlags: [
      '--headless=new',
      '--disable-gpu',
      '--no-sandbox',
      '--disable-extensions',
      '--remote-debugging-port=9222'
    ]
  });
  return chrome;
}

async function runAudit(url) {
  const chrome = await startChrome();
  try {
    // Connect Playwright to the running Chrome via CDP
    const browser = await playwright.chromium.connectOverCDP('http://127.0.0.1:9222');
    const contexts = browser.contexts();
    // Use a fresh context for isolation
    const context = await browser.newContext({ viewport: { width: 1200, height: 900 } });
    const page = await context.newPage();

    // Navigate and wait for hydration — customize wait conditions per site
    await page.goto(url, { waitUntil: 'networkidle' });
    // Optional: wait for a specific selector that indicates content rendered
    // await page.waitForSelector('article');

    // Extract structured data (JSON-LD) and basic page signals
    const extracted = await page.evaluate(() => {
      const jsonLd = Array.from(document.querySelectorAll('script[type="application/ld+json"]'))
        .map(s => s.innerText.trim());
      const meta = {
        title: document.title || null,
        h1: (document.querySelector('h1') && document.querySelector('h1').innerText) || null,
        canonical: (document.querySelector('link[rel="canonical"]') && document.querySelector('link[rel="canonical"]').href) || null,
        description: (document.querySelector('meta[name="description"]') && document.querySelector('meta[name="description"]').content) || null
      };
      // Collect candidate entity mentions from headings and first paragraphs
      const candidates = Array.from(document.querySelectorAll('h1,h2,h3,p'))
        .slice(0, 8)
        .map(el => el.innerText.trim())
        .filter(Boolean);
      return { jsonLd, meta, candidates };
    });

    // Run Lighthouse programmatically against the same debug port
    const lhOptions = { port: 9222, output: 'json', logLevel: 'info' };
    const runnerResult = await lighthouse(url, lhOptions);

    const reportJson = runnerResult.lhr; // Lighthouse result object

    // Aggregate findings into a compact audit object
    const audit = {
      url,
      timestamp: new Date().toISOString(),
      lighthouse: {
        categories: reportJson.categories,
        audits: {
          'first-contentful-paint': reportJson.audits['first-contentful-paint'],
          'largest-contentful-paint': reportJson.audits['largest-contentful-paint'],
          'interactive': reportJson.audits['interactive']
        },
        score: reportJson.categories.performance.score
      },
      structuredData: extracted.jsonLd,
      meta: extracted.meta,
      entityCandidates: extracted.candidates
    };

    // Persist outputs
    await fs.outputJson(`./reports/${encodeURIComponent(url)}.audit.json`, audit, { spaces: 2 });
    await fs.outputFile(`./reports/${encodeURIComponent(url)}.lhr.json`, JSON.stringify(reportJson, null, 2));

    // Optionally write Lighthouse HTML
    const htmlReport = runnerResult.report; // if output: 'html' was set
    if (htmlReport) {
      await fs.outputFile(`./reports/${encodeURIComponent(url)}.lighthouse.html`, htmlReport);
    }

    await context.close();
    await browser.close();
    await chrome.kill();

    return audit;
  } catch (err) {
    await chrome.kill();
    throw err;
  }
}

// Example runner
(async () => {
  const url = process.argv[2] || 'https://example.com';
  const result = await runAudit(url);
  console.log('Audit saved for', url);
})();

Notes on the example

We start a single Chromium using chrome-launcher to expose a stable CDP port (9222), allowing both Playwright and Lighthouse to connect.
Use page.goto with site-specific waits (networkidle, specific selectors) to ensure the content you need is rendered before extraction.
Saving both Lighthouse JSON and the audit JSON lets you build dashboards and perform diffs over time.

Extending the spider for scale

The single-worker example is intentionally simple. For crawl-scale audits (thousands of URLs) add a queue, concurrency controls, retries, and proxy rotation. Key patterns:

Use p-queue or BullMQ for persistent queues and rate limiting.
Open a pool of browser instances and reuse browser contexts to reduce startup overhead.
Rotate proxies per context when crawling multiple domains to avoid IP blocks.
Monitor resource usage: Lighthouse is CPU/memory heavy — schedule audits during off-peak CI windows or use spot instances.

Example: concurrency sketch

const PQueue = require('p-queue');
const queue = new PQueue({ concurrency: 4 }); // tune by instance size

for (const url of urls) {
  queue.add(() => runAudit(url).catch(err => { console.error('Audit failed', url, err); }));
}

await queue.onIdle();

Capturing richer entity signals

Structured data is the first-class signal. But modern entity SEO combines multiple cues. Here are recommended extractions to add to the worker:

Schema types: extract @type values from JSON‑LD and microdata.
sameAs / identifier links: these tie pages to canonical entity profiles (Wikipedia, Wikidata, social accounts).
Primary entity context: mainEntityOfPage, about, and itemReviewed.
Prominent text candidates: headings and first N sentences — pass these to an NER pipeline or LLM for canonicalization (optional).
Cross-page entity graph: record edges such as (page → sameAs → external ID) to build a site-level entity map.

For production, offload NER and entity linking to a specialized service (spaCy, Hugging Face models, or an LLM with an entity-resolve runner). Save results as structured entity records so product and SEO teams can prioritize disambiguation fixes.

Handling bot detection and anti-scraping

In 2026, websites increasingly fingerprint headless browsers. Recommended mitigations:

Use Playwright's connectOverCDP with a real Chromium binary (not minimal headless) to reduce detectable differences.
Rotate user agents and viewport sizes per context.
Use residential proxies or Egress IP pools for scale; respect robots.txt and rate limits.
Implement randomized human-like interactions when testing anti-bot failures: small mouse moves, scroll, or timed delays (only where legal and ethical).
Monitor for CAPTCHAs and implement an alerting path (manual review or human-in-the-loop) rather than bypassing them silently.

Quality gates and actionable outputs

An audit is only valuable if it produces prioritized, traceable action items. Map raw findings to fixes and confidence levels:

High priority: Missing required structured data (product price, availability) or canonical mismatch between rendered and server HTML.
Medium priority: Lighthouse performance regressions, missing meta description on high-value pages, conflicting hreflang tags.
Low priority: Minor accessibility flags, optional structured data improvements.

Export audits to CSV with columns: url, status, lighthouse.performance, missing_jsonld, primary_schema_types, top_entity_candidates, suggested_fixes. Attach Lighthouse HTML to tickets for engineers.

2026 trends you should account for

Search Generative Experience (SGE) matured: Google and other engines use entity graphs and structured data heavily. Capture entity linking signals to surface in SGE features.
LLM-assisted SEO is now part of the toolchain: use local or private LLMs to normalize extracted entities and derive content gaps at scale.
Privacy-driven browser changes introduced more headless detection vectors — using full browser binaries and realistic contexts reduces false positives.
Audit-as-code: integrate audits into CI/CD pipelines so developers catch regressions before deploy.

Real-world checklist for running end-to-end audits

Define target URL set and sitemap-driven prioritization for business-critical templates.
Configure Playwright contexts with proxy/user-agent/viewport rotation.
Set rendering wait rules per template (networkidle, selector, or custom JS event).
Collect JSON-LD, microdata, meta tags, and main DOM text snippets.
Run Lighthouse on the same browser session to capture accurate metrics.
Normalize entity candidates via a linking service and persist canonical IDs.
Generate prioritized tickets with links to Lighthouse HTML and evidence snippets.
Schedule recurring audits and diff results to detect regressions.

Security, compliance, and legal notes

Automated crawling and rendering can hit legal and ethical boundaries. Always:

Respect robots.txt and site-specific crawl-rate instructions.
Avoid automating actions that impersonate users or bypass authentication without explicit authorization.
Consult your legal team when scraping third-party sites or processing personal data obtained via crawls.

Actionable takeaways

Start with a small, repeatable worker that uses Playwright to render and Lighthouse to audit — the example above is production-ready with modest enhancements.
Extract structured data and entity candidates on every page; structured data is non-negotiable for modern ranking features.
Scale with queues and pooled browsers, and keep Lighthouse jobs scheduled and monitored to catch regressions.

"Audits that don't render JavaScript are blind to the signals search engines prioritize in 2026 — make rendering and entities a first-class part of your audit pipeline."

Next steps — integrate and iterate

Ready to ship? Start by running the sample worker against your top 50 landing pages. Validate the extracted structured data and principal entity candidates with your SEO and content teams. Then add automation: schedule weekly audits for high-value templates and create alerts for critical regressions (missing schema, canonical changes, major Lighthouse score drops).

Final checklist before deployment

Credential and proxy management configured
Storage and retention policy for Lighthouse artifacts
Alerting for CAPTCHA/blocks and failed audits
Pipeline to convert audit findings into tickets for dev/SEO teams

Call to action

Build this spider as a central part of your technical SEO workflow: clone a starter repo, run it against a sample of pages, and hook the outputs into your issue-tracker. If you want a ready-made starter, download the minimal Playwright+Lighthouse spider, customize the rendering waits for your templates, and schedule a nightly audit. Ship fewer surprises — catch regressions in CI, not in search traffic.

Unknown

Contributor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.