How to Find the Hidden JSON API Behind a JavaScript Website

You download a product page with requests, print the HTML, and the price is not there. Neither is the title, the rating, or the stock count. The page looked fine in your browser a second ago. So where did the data go?

It went into a second request. Modern sites ship a near-empty HTML shell, then fetch the real content as JSON from an internal API and paint it in with JavaScript. Your HTTP client stops after the first step. The browser keeps going.

The good news: that internal API is usually reachable directly. If you can find it, you skip the whole rendering problem. No headless browser, no waiting for the DOM, no parsing brittle HTML. You get clean JSON that is often better structured than anything on the page. This guide shows you how to find it, how to replicate the request, and what to do when the site fights back.

Why the Data Is Not in the HTML

Most interactive sites are single-page applications built with React, Vue, Angular, or a framework like Next.js or Nuxt. When you load the page, the server returns a skeleton. Then client-side code runs in the browser and calls one or more backend endpoints for the actual data.

You can confirm this in two seconds. Right-click the page and choose “View Page Source” (not “Inspect”). That shows the raw HTML your HTTP client would receive. If you see an empty <div id="root"></div>, a wall of <script> tags, or a message asking you to enable JavaScript, the content is loaded later. The data you want lives behind an API call, not in this document.

There are three common places that data comes from:

• XHR/Fetch calls to a JSON or GraphQL endpoint after the page loads.
• Inline JSON embedded in the initial HTML, often in a <script id="__NEXT_DATA__" type="application/json"> block or a window.__INITIAL_STATE__ = {...} assignment.
• A mix: some data inline, the rest fetched as you scroll or click.

Your job is to figure out which one you are dealing with, then pull from the source instead of the rendered page.

Step 1: Open the Network Tab and Filter for Fetch/XHR

Open the page in Chrome or Firefox, open DevTools (F12), and go to the Network tab. Reload the page with the tab open so it captures everything from the start.

You will see dozens of entries: images, fonts, CSS, tracking pixels. Ignore them. Click the Fetch/XHR filter to show only the requests JavaScript made in the background. This is where API calls live.

Now interact with the page the way a user would. Scroll down, click “load more”, open a product, change a filter. Watch new rows appear in the Network tab as you do. Each row is a request the site made on your behalf. One of them is carrying your data.

To find the right one, click a request and look at the Response or Preview tab. You are hunting for a response that contains the values you saw on the page: the price, the list of items, the review text. When you see JSON with your fields in it, you have found the endpoint.

A few tips that save time:

• Sort by response size. Data endpoints are usually larger than pings and config calls.
• Type a known value into the Network filter box (a product name, a price). Chrome can search across response bodies with the search panel (Ctrl+Shift+F inside DevTools).
• Look at the request name and path. Endpoints like /api/v2/products, /graphql, or /_next/data/...json are strong signals.

Step 2: Read the Request Before You Copy It

Once you have the endpoint, click it and study the request itself, not just the response. You need to reproduce it exactly enough to get the same answer.

Check these parts under the Headers tab:

• Method and URL. Is it GET or POST? Note every query parameter. Parameters like page, limit, offset, cursor, and sort are your controls for pagination and ordering.
• Request headers. Many APIs check for headers the browser sends automatically. Common ones that matter: Accept: application/json, a Referer, an X-Requested-With, and sometimes a custom header like X-Api-Key or X-Client-Version.
• Request body. For POST and GraphQL calls, the body holds the query and variables. Copy it as-is to start.
• Cookies and tokens. Some endpoints need a session cookie or a bearer token that the site issued on page load.

The fastest way to get a working baseline is “Copy as cURL”. Right-click the request, choose Copy > Copy as cURL, and paste it into your terminal. If it returns the same JSON, you have a faithful reproduction. From there you can strip it down.

Step 3: Replicate It in Code and Trim the Fat

Start from the full copied request, confirm it works, then remove headers one at a time until it breaks. What remains is the minimum you actually need. This keeps your scraper simple and less fragile.

Here is a minimal Python example calling a typical JSON endpoint:

import requests

url = "https://example.com/api/v2/products"
params = {"category": "shoes", "page": 1, "limit": 48}
headers = {
    "Accept": "application/json",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Referer": "https://example.com/category/shoes",
}

resp = requests.get(url, params=params, headers=headers, timeout=20)
data = resp.json()

for item in data["products"]:
    print(item["title"], item["price"])

For a GraphQL endpoint, send a POST with the query and variables you copied:

import requests

payload = {
    "query": "query Products($page:Int!){ products(page:$page){ title price } }",
    "variables": {"page": 1},
}
resp = requests.post("https://example.com/graphql", json=payload, timeout=20)
print(resp.json()["data"]["products"])

Node with fetch looks almost identical:

const res = await fetch("https://example.com/api/v2/products?page=1&limit=48", {
  headers: { Accept: "application/json", Referer: "https://example.com/category/shoes" },
});
const data = await res.json();
data.products.forEach((p) => console.log(p.title, p.price));

Step 4: Handle Pagination From the API, Not the Page

This is where direct API access pays off. The page might show a “Load More” button or infinite scroll, but the API almost always exposes a clean pagination control. Look at how the parameters change between the first and second fetch in the Network tab.

You will usually see one of three patterns:

• Page numbers: ?page=1, ?page=2. Loop until you get an empty list.
• Offset and limit: ?offset=0&limit=48, then ?offset=48. Increment by the limit.
• Cursor based: each response includes a nextCursor or endCursor token you pass into the next request. Follow it until it is null.

Cursor pagination is common on large feeds and is the most reliable of the three. It also tends to sidestep the deep-pagination caps that break page-number crawlers on big catalogs.

Step 5: When the Inline JSON Is Enough

Sometimes you do not even need a second request. Framework sites often embed the full dataset in the first HTML response. View the page source and search for __NEXT_DATA__, __NUXT__, __INITIAL_STATE__, or application/ld+json. If your data is in there, parse the HTML once, pull the JSON out of that script tag, and you are done. No API replay needed.

import json, re, requests

html = requests.get("https://example.com/product/123").text
match = re.search(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.S)
state = json.loads(match.group(1))
print(state["props"]["pageProps"]["product"]["price"])

This is fast and stable because you are reading the same data the framework used to build the page. Structured data in application/ld+json is especially handy for products, articles, and reviews.

When the API Fights Back

Not every endpoint hands over data to a plain script. The same anti-bot layers that protect the HTML page often sit in front of the API too. Signs you have hit one:

• A 403 or 429 even though your browser loads the endpoint fine.
• A challenge or CAPTCHA HTML body instead of JSON.
• A short-lived token, signed parameter, or timestamp that expires in seconds.
• A 200 response that returns an empty or decoy payload. Always assert on the shape of the JSON, not just the status code.

Some of this you can solve yourself. Rotate a realistic User-Agent, send the headers the browser sends, and reuse a session cookie you captured once rather than warming a fresh one every call. A library like curl_cffi helps when the block is based on your TLS handshake rather than your headers, because it mimics a real browser’s fingerprint.

But when an endpoint requires a signed token generated by obfuscated client-side JavaScript, or the whole domain sits behind an aggressive anti-bot vendor, replaying the request by hand becomes a losing game of reverse engineering that breaks every time the site ships a new build. That is the point where a scraping API earns its place: you send the target URL, it handles the browser rendering, fingerprinting, and proxy rotation, and you get the response back. ScrapeUnblocker is built for exactly that case, and its parsed endpoints can hand you structured JSON so you still skip the HTML parsing step.

FAQ

How do I know if a site uses a hidden API at all? View the raw page source. If the values you want are missing and you see an empty container or an “enable JavaScript” notice, the data is fetched separately. Confirm by watching the Fetch/XHR tab in DevTools as the page loads.

Is calling a site’s internal API legal? You are calling the same endpoint your browser calls, so it is a public request. That said, respect the site’s terms of service, robots guidance, and rate limits, and only collect publicly visible data. Legal footing depends on jurisdiction and use case, so treat this as an engineering answer, not legal advice.

Why use the API instead of a headless browser? Speed and stability. An API call takes milliseconds and returns clean, typed data. A headless browser spins up a full page render for every request and gives you HTML you still have to parse. Use the API when you can, and fall back to rendering only when you must.

The API needs a token I cannot generate. Now what? If the token comes from obfuscated JavaScript, capture a fresh page load to read the current token, or render the page once to obtain it and reuse the session. If that is too fragile to maintain, route the request through a scraping API that renders and returns the result for you.

Wrapping Up

The rendered page is the slow, brittle surface of a website. Underneath it is a clean data pipe the site built for its own frontend. Learning to read the Network tab, replicate the request, and follow the pagination turns most “JavaScript-heavy, impossible to scrape” sites into a simple JSON fetch.

Start with View Source and the Fetch/XHR filter on your next target. When you hit an endpoint that is locked behind serious anti-bot defenses, that is the moment to reach for a tool like ScrapeUnblocker instead of fighting the fingerprint by hand.