Validate Web Scraping Data with JSON Schema: Rules, Fixtures, and Quality Checks
json-schemadata-validationweb-scrapingdata-qualityetlschema-management

Validate Web Scraping Data with JSON Schema: Rules, Fixtures, and Quality Checks

WWeb Dev Toolbox Editorial Team
2026-11-21
8 min read

Learn how to validate scraped records with JSON Schema, quarantine failures, test fixtures, and protect databases from malformed web data.

Scraping a page is only the first step; the extracted records must also be trustworthy enough for the next system to use. This guide shows how to validate scraped data with JSON Schema, define required fields and data types, catch malformed values, report failures, and separate quarantine records from production output. Use the checklist whenever a source changes, a new crawler is added, or a scheduled import begins producing unreliable results.

Overview

JSON Schema gives a scraping pipeline an explicit contract. Instead of allowing every record through and discovering problems later in a database or report, you describe what a valid record should contain. The schema can specify required properties, string formats, numeric limits, arrays, nested objects, and whether unexpected fields are allowed.

Consider a product record. A minimal contract might require a non-empty product name, a URL, a numeric price, an ISO-style date, and an array of categories. An individual source may omit a value, change a label, or return a placeholder such as “N/A.” Validation makes that difference visible before ingestion.

A schema does not prove that a value is factually correct. A URL can be well-formed but point to the wrong page, and a price can be numeric but belong to a different currency. Treat schema validation as one layer in a wider quality process that can also include normalization, duplicate detection, source checks, and business rules. For related cleanup work, see Clean and Normalize Web Scraping Data: Dates, Prices, Text, and Missing Values.

Useful JSON Schema keywords

KeywordPurposeScraping example
typeSets the expected data type.price must be a number.
requiredLists properties that must exist.name and source_url cannot be omitted.
propertiesDefines rules for object fields.Applies a pattern or format to each field.
itemsDefines rules for array members.Every category must be a string.
minLengthRejects empty or too-short strings.A title must contain at least one character.
patternChecks a string against a regular expression.Restricts an identifier to an expected shape.
formatRequests a recognized format check.Checks values intended to be URLs or dates.
additionalPropertiesControls unexpected fields.Flags fields introduced by a source change.
minimum and maximumSets numeric boundaries.Prevents a negative price.

Checklist by scenario

1. Define the record contract before extraction

  • List each field the downstream database, report, or API actually needs.
  • Choose a stable internal name for every field instead of copying inconsistent labels from the source.
  • Decide which fields are required, optional, nullable, or conditionally required.
  • Document units and conventions, such as a decimal price, a currency code, or a UTC timestamp.
  • Record the source URL and retrieval time where traceability matters.

Keep the first schema focused. If a field is not used and is difficult to extract reliably, consider leaving it out of the initial contract. You can add it in a controlled schema revision later.

2. Validate a product-style record

The following template illustrates common rules. Adjust the fields to match the source and your storage model:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Scraped product",
  "type": "object",
  "additionalProperties": false,
  "required": ["name", "source_url", "price", "categories"],
  "properties": {
    "name": {"type": "string", "minLength": 1},
    "source_url": {"type": "string", "format": "uri"},
    "price": {"type": "number", "minimum": 0},
    "currency": {"type": "string", "pattern": "^[A-Z]{3}$"},
    "available_on": {"type": ["string", "null"], "format": "date"},
    "categories": {"type": "array", "items": {"type": "string"}, "uniqueItems": true}
  }
}

Use nullable fields deliberately. A missing optional field and an explicit null value may mean different things in your database. If the extractor emits empty strings, normalize them before validation or make the schema account for them explicitly. Do not silently turn every failed value into null; that can hide selector or parsing failures.

3. Validate article or content records

Content pipelines commonly need a title, canonical URL, body text, publication state, and extracted links. Add limits that protect downstream systems, but avoid arbitrary restrictions that reject legitimate content. For example, a title can require a minimum length while body length may be checked as a warning rather than a hard failure.

When extracting structured data such as JSON-LD, keep the original object separately from the normalized record. This preserves evidence when a source publishes multiple objects or when your mapping logic needs review. The guide Scrape Structured Data from Web Pages: JSON-LD, Open Graph, and Microdata Compared covers the extraction side of that workflow.

4. Add validation to a Python pipeline

A Python scraper can validate each normalized record with a JSON Schema library before writing to a database:

from jsonschema import Draft202012Validator

validator = Draft202012Validator(schema)

for record in records:
    errors = sorted(validator.iter_errors(record), key=lambda error: list(error.path))
    if errors:
        quarantine(record, [
            {"path": list(error.path), "message": error.message}
            for error in errors
        ])
    else:
        write_to_staging(record)

Validate after parsing and normalization, not against raw HTML fragments. A value such as “$19.99” may need to become a numeric decimal before it can satisfy a numeric rule. Keep the raw value in an audit field if you need to investigate parsing decisions.

5. Add validation to a JavaScript or Node.js pipeline

For a Node.js scraper, use a validator compatible with the schema dialect you selected. The shape of the workflow remains the same: compile the schema once, validate normalized records, capture errors, and write only valid records to the next stage.

const validate = ajv.compile(schema);

for (const record of records) {
  if (validate(record)) {
    await writeToStaging(record);
  } else {
    await quarantine(record, validate.errors);
  }
}

Keep the validator configuration consistent between development and scheduled jobs. Differences in format handling or strictness can produce confusing results when a fixture passes locally but fails in production.

6. Separate valid, rejected, and warning outcomes

A useful pipeline does not reduce every result to pass or fail. Use at least three outcomes:

  • Valid: Meets the schema and can proceed to staging or ingestion.
  • Rejected: Violates a required rule, has an invalid type, or cannot be safely interpreted.
  • Warning: Passes the structural contract but needs review, such as an unexpectedly short description or a new optional field.

For rejected records, store the record or a protected copy, source identifier, retrieval time, schema version, and structured error paths. A message such as price: expected number is more useful than a generic “validation failed.” Apply retention and access controls appropriate to the data you collect.

What to double-check

  • Dates: Decide whether dates include a time zone and whether the extractor should convert them to a common representation. A format check alone may not resolve ambiguous dates.
  • URLs: Validate the URL shape, then separately check canonicalization, allowed schemes, redirects, and whether the URL belongs to the expected source.
  • Numbers: Confirm decimal separators, currency, units, and whether the source uses a range such as “10–20.” Do not coerce a range into a single number without a defined rule.
  • Arrays: Check whether an empty array is valid, whether duplicate values matter, and whether array members need their own schema.
  • Optional fields: Distinguish absent, null, empty, and placeholder values. These states often have different reporting meanings.
  • Unexpected fields: Use additionalProperties: false when schema drift should stop the record. Use a more permissive approach when the source contains harmless fields that you intentionally preserve.
  • Database boundaries: Validate before production writes, but also keep database constraints for critical invariants. Application validation and database validation protect against different failure paths.
  • Schema versions: Store the schema identifier or version with each batch. This makes historical records easier to interpret after a contract changes.

Run representative fixtures through the validator: a complete valid record, each missing required field, wrong data types, empty strings, null values, malformed URLs, boundary numbers, unexpected fields, and a record with several simultaneous errors. Pair this with regression tests for selectors and parsers; see How to Test Web Scrapers with Fixtures, Mocks, and Regression Suites.

Common mistakes

Validating raw data too early

Raw page text is rarely in the final shape required by a schema. Extract, normalize, and map the record first. Then validate the representation that will actually be stored or transmitted.

Using type checks as a substitute for business rules

A string can be empty, a date can be implausible for the source, and a positive price can still be wrong. Add separate checks for domain expectations, such as a permitted currency or a reasonable change from the previous observation.

Dropping invalid records without an audit trail

Silent rejection makes a scraper appear successful while reducing coverage. Quarantine failures with enough context to reproduce the problem, and report error counts by field and source.

Making every field required

Overly strict schemas create noise when a source legitimately omits optional information. Require only what downstream processing truly depends on, and use warnings for lower-risk gaps.

Changing the schema without changing fixtures

Every schema revision should update valid and invalid fixtures, migration logic, and monitoring expectations. A contract is maintained through tests, not just a JSON file.

When to revisit

Review the schema before seasonal planning cycles if the source changes its catalog, publishing pattern, or expected volume. Revisit it whenever a selector, parser, browser automation step, API mapping, database column, or downstream report changes. A new website should receive its own sample fixtures before being added to a shared ingestion path.

Set a practical review trigger for quality signals: a sudden rise in missing required fields, a new validation error path, an unusual increase in nulls, a drop in valid-record volume, or reports showing malformed values. These signals can indicate source markup changes, an expired session, a parsing bug, or a legitimate change in the source data.

At each review, compare a recent sample with the current schema, inspect quarantined records, confirm that normalization rules still match storage requirements, and decide whether the change is backward-compatible. If the contract must change, publish a new version, update fixtures, run the pipeline against a staging destination, and record the migration decision.

Action checklist: define the record contract; normalize values; validate with a versioned schema; test valid, invalid, and boundary fixtures; quarantine failures; monitor error paths; and schedule the next review when the source or workflow changes. That sequence turns JSON Schema from a one-time check into a dependable quality gate for web scraping data.

Related Topics

#json-schema#data-validation#web-scraping#data-quality#etl#schema-management
W

Web Dev Toolbox Editorial Team

Technical Editorial Team

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.