How to Extract Schema.org and JSON-LD Data From Any Web Page

You write CSS selectors for a product page. They work for two weeks. Then the site ships a redesign, the price moves into a new <span>, and your scraper quietly returns None for every row.

Meanwhile, the same page has been carrying its name, price, currency, stock status and rating in a clean JSON block the whole time. It is there for Google, not for you, but nothing stops you from reading it. This guide shows how to find that Schema.org data, extract it with Python, turn it into a clean record, and deal with the cases where it lets you down.

What Structured Data Is and Why Sites Publish It

Schema.org is a shared vocabulary for describing things on the web: Product, Offer, JobPosting, Recipe, Article, Event, LocalBusiness and hundreds more. Search engines read it to build rich results, the star ratings, prices and stock labels you see under some search results. Sites that care about search traffic publish it on their most important pages.

It comes in three syntaxes:

• JSON-LD. A <script type="application/ld+json"> block with a JSON document inside. This is the most common format today and the one Google recommends.
• Microdata. Attributes such as itemscope, itemtype and itemprop spread across the visible HTML. Common on older shops and templates.
• RDFa. Similar idea with vocab, typeof and property attributes. Less common.

Many pages also carry Open Graph meta tags (og:title, og:image, og:price:amount). They are not Schema.org, but they are a handy fallback for titles and images.

Why does this matter for scraping? Structured data is the most stable part of a page. A redesign changes class names, but the SEO team rarely breaks the markup that feeds their rich results. A parser built on it survives layout changes that kill selector-based scrapers.

How to Find It on a Page

Open the page, view the source (not the DevTools element tree, which shows the DOM after JavaScript ran) and search for:

• application/ld+json for JSON-LD blocks
• itemprop= for Microdata
• property="og: for Open Graph tags

A page often has several JSON-LD blocks: one for the organization, one for breadcrumbs, one for the product. To see everything a page publishes in one view, paste the URL into the Schema Markup Validator. It lists every item it finds, in every syntax.

Extract JSON-LD With Python

You need requests and beautifulsoup4. The helper below collects every JSON-LD node on a page and handles two shapes that trip up first attempts: a block that holds a list of objects, and a block that wraps everything in an @graph array.

import json
import requests
from bs4 import BeautifulSoup

def jsonld_nodes(html):
    soup = BeautifulSoup(html, "html.parser")
    nodes = []
    for tag in soup.find_all("script", type="application/ld+json"):
        raw = tag.string or tag.get_text()
        if not raw or not raw.strip():
            continue
        try:
            # strict=False accepts raw newlines inside strings, a common mistake
            doc = json.loads(raw, strict=False)
        except json.JSONDecodeError:
            continue
        for item in doc if isinstance(doc, list) else [doc]:
            if isinstance(item, dict) and "@graph" in item:
                nodes.extend(item["@graph"])
            else:
                nodes.append(item)
    return [n for n in nodes if isinstance(n, dict)]

def has_type(node, wanted):
    types = node.get("@type", [])
    return wanted in (types if isinstance(types, list) else [types])

Note that @type can be a string or a list. "@type": ["Product", "Car"] is valid, so always check membership instead of comparing strings.

Or Read Every Syntax at Once With extruct

If you scrape many different sites, some will only use Microdata or RDFa. The open-source extruct library reads all of them in one call:

import extruct
from w3lib.html import get_base_url

def all_structured_data(html, url):
    return extruct.extract(
        html,
        base_url=get_base_url(html, url),
        syntaxes=["json-ld", "microdata", "rdfa", "opengraph"],
        uniform=True,  # Microdata and Open Graph in the same shape as JSON-LD
        errors="ignore",
    )

With uniform=True, Microdata items come back with @type and the same property names as JSON-LD, so the same downstream code can handle both.

One catch: extruct parses JSON-LD per page, not per block. If one block on the page is broken, errors="ignore" drops all the JSON-LD for that page, including the valid product block. A good split is the jsonld_nodes helper above for JSON-LD, and extruct for Microdata, RDFa and Open Graph.

Turn a Product Node Into a Clean Record

Finding the Product node is the easy part. The Schema.org spec allows a lot of variation, and real sites use all of it:

• offers can be a single Offer, a list of offers, or an AggregateOffer with lowPrice and highPrice.
• price can be a number, a string like "1299.00", or a string with a thousands separator. Sometimes it hides in a nested priceSpecification.
• availability is usually a URL such as https://schema.org/InStock, sometimes with http, sometimes just InStock.
• brand can be a plain string or an object with a name.
• image can be a string, a list, or an ImageObject with a url.

A small normalizer absorbs all of that:

import re

def first(value):
    if isinstance(value, list):
        return value[0] if value else None
    return value

def name_of(value):
    value = first(value)
    return value.get("name") if isinstance(value, dict) else value

def to_number(value):
    if isinstance(value, (int, float)):
        return float(value)
    text = re.sub(r"[^\d.,]", "", str(value or ""))
    if "," in text and "." in text:
        # Whichever separator comes last is the decimal one
        if text.rfind(",") > text.rfind("."):
            text = text.replace(".", "").replace(",", ".")
        else:
            text = text.replace(",", "")
    elif "," in text:
        decimals = text.split(",")[-1]
        text = text.replace(",", ".") if len(decimals) <= 2 else text.replace(",", "")
    try:
        return float(text)
    except ValueError:
        return None

def product_record(node):
    offer = first(node.get("offers")) or {}
    spec = first(offer.get("priceSpecification")) or {}
    if has_type(offer, "AggregateOffer"):
        price = offer.get("lowPrice") or offer.get("price")
    else:
        price = offer.get("price") or spec.get("price")
    image = first(node.get("image"))
    rating = node.get("aggregateRating") or {}
    return {
        "name": node.get("name"),
        "sku": node.get("sku"),
        "brand": name_of(node.get("brand")),
        "price": to_number(price),
        "currency": offer.get("priceCurrency") or spec.get("priceCurrency"),
        "availability": str(offer.get("availability") or "").rstrip("/").split("/")[-1] or None,
        "image": image.get("url") if isinstance(image, dict) else image,
        "rating": to_number(rating.get("ratingValue")),
        "review_count": to_number(rating.get("reviewCount") or rating.get("ratingCount")),
    }

url = "https://www.example.com/product/123"
html = requests.get(url, timeout=30).text
products = [product_record(n) for n in jsonld_nodes(html) if has_type(n, "Product")]
print(products)

The same pattern works for other types. A JobPosting has title, hiringOrganization, jobLocation and baseSalary. A Recipe has recipeIngredient and recipeInstructions. A RealEstateListing or Car brings its own properties. Write one small mapping function per type you care about, not per site.

Where Structured Data Lets You Down

Structured data is the best first move, not a complete answer. Plan for these cases.

The Spec Sheet Is Not in the Markup

Sites publish what search engines reward: name, price, rating, availability. The detailed specs a buyer reads, such as dimensions, materials, engine power or floor area, usually live only in an HTML table or a definition list. You can read those generically, without per-site selectors, by collecting label and value pairs:

def spec_table(html):
    soup = BeautifulSoup(html, "html.parser")
    specs = {}
    for row in soup.select("table tr"):
        cells = row.find_all(["th", "td"])
        if len(cells) == 2:
            specs[cells[0].get_text(" ", strip=True)] = cells[1].get_text(" ", strip=True)
    for dt in soup.find_all("dt"):
        dd = dt.find_next_sibling("dd")
        if dd:
            specs[dt.get_text(" ", strip=True)] = dd.get_text(" ", strip=True)
    return specs

Keep the labels as the page prints them. If you scrape the same site in several languages, map the labels to your own field names afterwards (“Kilometerstand” and “Mileage” are the same field).

The JSON-LD Is Added by JavaScript

Some sites inject their structured data client-side, for example through a tag manager. The raw HTML from requests then has no JSON-LD at all, while the validator, which renders the page, shows it fine. Render the page with a browser and read the scripts from the DOM:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto(url, wait_until="networkidle")
    blocks = page.locator('script[type="application/ld+json"]').all_text_contents()
    browser.close()

The Data Is Stale or Belongs to Another Variant

Google’s guidelines require markup to match the visible content, but templates and caches drift. On a product with sizes or colors, the JSON-LD may describe the default variant, or list every variant under a ProductGroup with hasVariant. Spot-check a sample of pages against what the browser shows before you trust the price field at scale.

The JSON Is Broken

Hand-built JSON-LD contains trailing commas, unescaped quotes and HTML comments. strict=False handles raw newlines. For the rest, a tolerant parser such as chompjs recovers many blocks that json.loads rejects. If a block still fails, log it and fall back to the next source.

There Is Nothing Because You Were Blocked

An empty result does not always mean the page has no markup. A bot challenge or a “403 dressed as 200” page has no JSON-LD either. Check for block signals before you conclude a site does not publish structured data. Our guide on detecting soft blocks covers the usual signs.

A Practical Extraction Order

For each page, try the sources from most to least stable and stop when you have the fields you need:

1. JSON-LD
2. Microdata and RDFa
3. Embedded framework state such as __NEXT_DATA__ (see how to find the hidden JSON behind a JavaScript website)
4. Open Graph meta tags for title, description and image
5. Spec tables and labelled values
6. CSS or XPath selectors, as the last resort

Record which source each field came from. When a site changes, you will know which layer moved.

FAQ

What is the difference between JSON-LD, Microdata and RDFa?

They are three ways to write the same Schema.org vocabulary. JSON-LD is a separate JSON block in a script tag. Microdata and RDFa are attributes on the visible HTML elements. The data model is the same, which is why extruct can return all three in one shape.

Is the price in JSON-LD always the price shown on the page?

Usually, because search engines expect markup to match visible content. It can lag after a price change, or describe a different variant than the one selected. Validate on a sample before you rely on it.

Do I need a headless browser to read structured data?

Only when the site injects it with JavaScript or blocks plain HTTP clients. Most JSON-LD is in the server-rendered HTML, so a simple request is enough.

Can I combine structured data with my own selectors?

Yes, and that is the most robust setup. Take every field you can from structured data, and use selectors only for the few fields it does not cover.

Let an API Do the Parsing

If you would rather not maintain extraction code for every page type, ScrapeUnblocker can return structured JSON instead of HTML. Add parsed_data=true to a /getPageSource request:

import requests

r = requests.post(
    "https://api.scrapeunblocker.com/getPageSource",
    params={"url": "https://www.example.com/product/123", "parsed_data": True},
    headers={"x-scrapeunblocker-key": "YOUR_API_KEY"},
    timeout=120,
)
print(r.json()["data"])

The response says what kind of page it was (product_detail, listing_detail, search_list and so on) and returns English snake_case fields with prices normalized to a value and a currency. Detail pages now also carry the page’s whole spec sheet as specifications, exactly as the page prints it, next to the Schema.org facts. When a page has nothing to extract, you get the rendered HTML back, so the parser from this guide still has something to work with. The parsed data guide shows the full response shape.