How to Extract Structured Data from Web Pages: JSON-LD, Microdata, and RDFa

Overview

Structured data is machine-readable information embedded in a page. It can describe entities such as products, articles, organizations, events, local businesses, recipes, or pages in a website hierarchy. Schema.org vocabulary is commonly expressed through three markup formats: JSON-LD, Microdata, and RDFa.

These formats can appear together on the same page, and a single page may contain several entities. For example, an article page might include a top-level Article, a nested Person for the author, a ImageObject for the image, and a Organization for the publisher. An extraction process should preserve those relationships rather than flattening every value into one undifferentiated list.

Start by deciding what the audit needs. A page inventory may only require the schema types present. A template comparison may also need fields such as name, url, image, author, datePublished, offers, or aggregateRating. A reliable extractor records both the normalized value and the original location so that every result can be traced back to the page.

For broader extraction work, separate markup discovery from browser automation. Static HTML may be enough for server-rendered pages, while client-rendered pages may require a browser workflow. The Playwright guide for JavaScript-rendered websites explains that distinction in a wider scraping context.

Checklist by scenario

Scenario 1: Inspect one page manually

1. Open the page source as well as the rendered DOM in browser developer tools.
2. Search the source for application/ld+json, itemscope, itemtype, typeof, and property.
3. Record every markup block, not only the first match.
4. Copy JSON-LD into a JSON formatter or validator before interpreting it. A formatted view makes arrays, nested objects, and syntax errors easier to see.
5. Note the page URL, canonical URL, schema type, entity identifier, and source format.
6. Compare important values with the visible page content and with the canonical version of the URL.

This manual pass is useful when troubleshooting a single template or verifying whether a deployment changed markup. It also establishes the expected output before you automate the process.

Scenario 2: Extract JSON-LD at scale

1. Fetch the page responsibly and retain the response URL, status information, and retrieval timestamp in your audit record.
2. Locate all script elements whose type is application/ld+json.
3. Parse each block independently so one malformed block does not hide valid blocks elsewhere on the page.
4. Accept common top-level shapes: one object, an array of objects, or an object containing an @graph array.
5. Preserve JSON-LD keys such as @context, @type, and @id during raw capture.
6. Normalize selected fields into audit columns only after the original object has been stored.

A minimal Python pattern using BeautifulSoup might look like this:

import json
from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
blocks = []

for node in soup.select('script[type="application/ld+json"]'):
    raw = node.string or node.get_text()
    try:
        blocks.append({"data": json.loads(raw), "error": None})
    except json.JSONDecodeError as error:
        blocks.append({"data": None, "error": str(error)})

This is discovery logic, not a complete Schema.org validator. It deliberately records parse errors for later review. For a fuller Python workflow covering requests, BeautifulSoup, and browser rendering, see how to scrape a website with Python.

Scenario 3: Extract Microdata

1. Find elements with itemscope and identify their itemtype.
2. For each item, collect descendants with itemprop until another relevant item scope changes the context.
3. Read values according to the element: content for meta elements, datetime for time elements, href for links, src for images, and visible text where appropriate.
4. When a property contains a nested itemscope, store it as an object or linked child rather than a plain string.
5. Keep repeated properties as arrays when the page supplies more than one value.

Microdata extraction is a tree-building task. A flat selector that collects every itemprop on the page can assign a nested property to the wrong parent, especially when a page contains several products, offers, or people.

Scenario 4: Extract RDFa

1. Identify resources using attributes such as typeof, about, and resource.
2. Read property values as predicates connected to the current subject.
3. Resolve the value from content, href, src, or element text according to the element and markup.
4. Track nested subjects and preserve the relationship between the parent and child resources.
5. Store the vocabulary or type context when it is provided through vocab or a fully qualified value.

RDFa is best represented internally as relationships rather than as a simple row of fields. If your final report is tabular, create a relationship-aware intermediate format first and flatten it only for the columns the report requires.

What to double-check

Field-mapping checklist

• Identity: Is there an @id, url, or other stable identifier, and does it correspond to the page or entity being audited?
• Type: Is @type, itemtype, or typeof present and captured without being overwritten by a second type?
• Canonical URL: Does the extracted entity URL agree with the page’s canonical link when one is available?
• Nested objects: Are authors, publishers, offers, images, and ratings retained as structured objects?
• Repeated values: Are multiple images, authors, categories, or offers represented as arrays rather than silently discarded?
• Value source: Did the extractor read the intended value from content, datetime, href, src, or visible text?
• Dates and numbers: Were date strings, currencies, prices, scores, and counts preserved before normalization?
• Provenance: Can each normalized field be traced to a format, selector, script block, or source element?

Duplicate and conflicting markup

Do not automatically treat duplicate markup as a single truth. A page may contain equivalent JSON-LD and Microdata, or it may contain an older block left by a template component. Compare entity identifiers, types, and important fields. If two blocks disagree, report the conflict and preserve both source records. A useful audit output includes the number of blocks, the formats detected, the types found, and a short list of conflicting fields.

Rendered versus original HTML

Capture both the initial response and the final rendered DOM when the site uses client-side rendering. A crawler that inspects only the initial response can miss markup inserted after scripts run; a browser-only process can make it harder to understand what was delivered initially. Label the source of every extraction result as response HTML, rendered DOM, or another clearly defined input.

Once extracted data is cleaned and validated, follow a repeatable storage process. The guide to web scraping data pipeline design covers normalization, validation, and storage decisions that also apply to structured-data audits.

Common mistakes

• Parsing only the first JSON-LD block: Pages can contain several scripts, including separate entities or graphs.
• Assuming valid JSON means valid structured data: JSON syntax and schema completeness are different checks.
• Flattening nested properties too early: This loses relationships such as which offer belongs to which product.
• Using visible text for every value: Dates, URLs, image sources, and machine-readable content may live in attributes.
• Discarding malformed blocks: Keep the raw text and parse error so template owners can investigate the source.
• Mixing page-level and entity-level URLs: Store the retrieval URL, canonical URL, and entity URL in separate fields.
• Declaring a page empty after a static fetch: Test whether the page requires rendering before concluding that no markup exists.
• Trusting a single validator result: Use validation as one signal, then inspect the source, relationships, duplicates, and visible content yourself.

For robust page collection, selector resilience and monitoring matter as much as parsing. Review the CSS and XPath selector guide when a redesign changes the surrounding HTML, and use scraper monitoring and data-quality checks to detect sudden extraction failures.

When to revisit

Revisit this workflow before a seasonal planning cycle, after a template redesign, when a new content type is introduced, or whenever an audit begins returning an unusual number of missing or conflicting fields. It is also worth reviewing when your collection method changes from direct HTTP requests to browser automation, when a new parser is introduced, or when the export schema is modified.

Keep a small regression set of representative pages: one page with a simple JSON-LD object, one with an @graph, one with nested Microdata, one with RDFa, one with multiple entities, and one with intentionally malformed markup. Run the extractor against this set whenever code or tools change. Compare block counts, types, identifiers, nested fields, parse errors, and canonical URLs rather than checking only whether a script completed.

Before closing an audit, use this final action list:

1. Confirm that the input was the correct URL and rendering state.
2. Capture every structured-data block and its original format.
3. Parse JSON-LD independently and record failures.
4. Build parent-child relationships for Microdata and RDFa.
5. Preserve raw data alongside normalized audit fields.
6. Check duplicates, conflicting values, canonical URLs, and visible-page alignment.
7. Export only after validation and document the extraction date, method, and known limitations.

This checklist makes structured data extraction repeatable: discover the markup, preserve its context, validate its relationships, and produce an audit record that can be compared again when the page or workflow changes.