Truncated HTML Responses: How to Catch a Scraped Page That Was Cut Off

Your scraper ran all night. Every request returned HTTP 200. The HTML files look fine when you open them: they start with <html> and end with </html>. But a handful of pages have 12 products instead of 48, and on some of them the embedded JSON throws a parse error. You fetch the same URL again by hand and get the full page.

That is a truncated response. The server, or something between you and the server, stopped sending the page halfway through, and nothing in your stack complained. It is rarer than a block, but harder to catch, because it looks almost exactly like success.

This guide explains how a 200 response can arrive cut off, which cases your HTTP client catches on its own, and how to check every page for completeness before you parse it.

What a Truncated Response Looks Like

Truncation is easy to mistake for a parser bug or a site change. The usual signs:

• Random gaps. The same page template gives full results most of the time and fewer items now and then.
• Embedded JSON that will not parse. json.loads() fails with “Unterminated string” or “Expecting ’,’ delimiter” on data that parsed fine yesterday.
• The fields at the bottom go missing first. Footer links, pagination, “related products” and anything else late in the document disappear, while the header and the first items are fine.
• A retry fixes it. The same URL a minute later returns the full page.
• Response sizes vary a lot. Most responses for one page type cluster around one size, with a scattered tail of smaller ones.

If those signs only appear on some requests and the missing part is always the end of the document, suspect truncation before you rewrite your selectors.

Why a 200 Can Arrive Cut Off

The status code is the first thing a server sends. By the time the body starts flowing, the 200 is already on the wire and cannot be taken back. Everything that goes wrong after that has to be signalled some other way.

HTTP/1.1 has three ways to mark where a response body ends:

• Content-Length. The server says up front how many bytes to expect.
• Chunked transfer encoding. The body arrives in pieces, and a final zero-length chunk says “done”.
• Connection close. No length at all. The body ends when the connection closes.

HTTP/2 and HTTP/3 have their own end-of-stream signal, but the idea is the same. Truncation happens when the body stops early and the framing either reports it as an error or, worse, reports it as a normal end.

Common causes:

• Streaming server rendering. Modern frameworks send the HTML as it renders. Headers and the 200 go out first, the rest follows. If rendering fails or times out halfway, the server can only end the stream. It cannot switch to a 500.
• Proxies, load balancers and gateways. Anything in the middle can hit a timeout or a size limit and close the stream. Some do it with a clean final chunk, so the client sees a normal, shorter response.
• Compression. A gzip body that stops halfway can decode to a shorter document without any error.
• Your own code. A size limit on a streamed read, a retry wrapper that returns the last partial body, or a headless browser that grabs the HTML before the page has finished.

Which Truncations Your Client Notices

Some truncations raise an error, some do not. Here is what happens when a server sends 200 and then stops early, tested with Python requests on urllib3 2.x, Node’s built-in fetch, and curl:

What the server did	Python requests	Node fetch	curl
Sent fewer bytes than Content-Length	ChunkedEncodingError (IncompleteRead)	TypeError: terminated	error 18, “transfer closed with N bytes remaining”
Dropped a chunked stream without the final chunk	ChunkedEncodingError (“Response ended prematurely”)	TypeError: terminated	error 18, “transfer closed with outstanding read data”
Ended a chunked stream cleanly after a short body	no error, status 200	no error	no error
Sent no length and closed the connection	no error, status 200	no error	no error
Sent a gzip body cut in half, framing clean	no error, shorter text	no error	no error

Two takeaways. First, the loud cases only help if you let them be loud. A broad except Exception: pass, or a retry helper that hands back whatever it got last, turns a clear error into silent partial data. Second, the bottom three rows look exactly like success at the HTTP level, so you need checks on the content itself.

One more detail: urllib3 2.x enforces Content-Length by default. The older 1.x line did not, so on a legacy stack even the first row can pass silently. Run pip show urllib3 if you are not sure.

Your Parser Hides the Damage

HTML parsers are built to be forgiving. Browsers have to render broken markup, so parsers like lxml and Python’s html.parser close every open element when the input ends. That is great for rendering. For a scraper it is a problem.

Feed a truncated document into BeautifulSoup and print it:

from bs4 import BeautifulSoup

cut = '<html><head><script id="__NEXT_DATA__" type="application/json">{"props":{"items":[{"id":1,"title":"A'

print(BeautifulSoup(cut, "lxml"))
# <html><head><script id="__NEXT_DATA__" type="application/json">{"props":{"items":[{"id":1,"title":"A</script></head></html>

The output ends in </script></head></html>, but the input never had those tags. Run soup.select("div.product-card") and you get fewer items, with no exception. Different parsers repair in different ways (html.parser drops the unfinished script contents entirely, lxml keeps them), but none of them tells you anything was missing.

Two rules follow from this:

1. Check completeness on the raw text, before parsing. After parsing, the evidence is gone.
2. Store the raw response, not a re-serialized copy. str(soup) or soup.prettify() adds the missing closing tags, so your archive will look complete when it was not. Re-serializing can also change the content: if a JSON response is passed through an HTML parser, characters like & come back out as &.

How to Detect a Truncated Page

No single check catches everything. Use a few cheap ones together.

1. Let the Transport Fail Loudly

Catch the specific exceptions and treat them as “incomplete, retry”, never as “done”:

from requests.exceptions import ChunkedEncodingError, ConnectionError

try:
    resp = session.get(url, timeout=(5, 30))
except (ChunkedEncodingError, ConnectionError) as e:
    # the body was cut off in transit - retry, do not parse
    ...

In Node, the error does not appear when fetch() resolves (you already have the 200 at that point). It appears when you call await res.text(), so put that call inside your try block too.

2. Check That the Document Ends Where It Should

On the raw text, look for </html> near the end:

def has_closing_html(raw: str) -> bool:
    return "</html>" in raw[-2048:].lower()

Look at the last few KB rather than the very last bytes, because many pages put scripts or comments after the closing tag. Also learn each site first: the HTML spec lets authors leave out </body> and </html>, so only rely on this check for sites whose healthy pages actually include it.

3. Validate Embedded JSON

This is the strongest check you have. JSON has no error recovery: a cut-off object can never parse. Framework state such as __NEXT_DATA__, application/ld+json blocks and inline app state are all good canaries:

import json
import re

NEXT_DATA = re.compile(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', re.S)

def next_data_ok(raw: str) -> bool:
    if '<script id="__NEXT_DATA__"' not in raw:
        return True  # this page does not use it
    m = NEXT_DATA.search(raw)
    if not m:
        return False  # opening tag present, closing tag never arrived
    try:
        json.loads(m.group(1))
        return True
    except json.JSONDecodeError:
        return False

If you already pull data out of these blocks (see how to find the hidden JSON behind a JavaScript website), a parse failure is your signal to retry, not a reason to skip the page.

4. Look for an End-of-Page Marker

Pick something that only appears near the bottom of a healthy page: the footer, the pagination block, a known final script. If it is missing, the page did not arrive in full. Combine it with an item count for listing pages. Our guide to detecting soft blocks covers count-based checks in more detail, since partial pages also appear there as an anti-bot tactic.

5. Track Response Sizes

Log the byte length of every response, grouped by page type. Truncation shows up as a cluster of small outliers. It is not proof for a single page, but a sudden rise in short responses tells you something changed upstream.

Put It Together

A small guard that runs before your parser, with a bounded retry:

import requests

def completeness_problems(raw, end_marker=None):
    problems = []
    if not has_closing_html(raw):
        problems.append("no </html> near the end")
    if end_marker and end_marker not in raw:
        problems.append(f"end marker {end_marker!r} missing")
    if not next_data_ok(raw):
        problems.append("__NEXT_DATA__ cut off or invalid")
    return problems

def fetch_complete(url, end_marker=None, attempts=3):
    last = []
    for _ in range(attempts):
        try:
            with requests.Session() as s:  # fresh connection per attempt
                resp = s.get(url, timeout=(5, 30))
        except (ChunkedEncodingError, ConnectionError) as e:
            last = [f"transfer broken: {e}"]
            continue
        problems = completeness_problems(resp.text, end_marker)
        if not problems:
            return resp.text
        last = problems
    raise RuntimeError(f"incomplete after {attempts} attempts: {last}")

Truncation is usually transient, so a retry on a fresh connection fixes most cases. If you route through proxies, switch to a different one on retry too. Keep the number of attempts small and log every failure with its reason. If the truncation rate climbs on one site, slow down: long, heavy pages are the ones that get cut when a server is under load.

Headless Browsers: Do Not Snapshot Too Early

With Playwright or Selenium, the closing-tag check is useless. page.content() serializes the current DOM, and a DOM is always well-formed. The risk here is taking the snapshot before the data has arrived, especially on streaming pages. Next.js App Router pages, for example, deliver their data in self.__next_f.push(...) script chunks as the stream comes in.

Wait for the content you need, not just for a load event:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto(url, wait_until="load")
    page.wait_for_selector("footer", state="attached", timeout=15000)
    page.wait_for_selector("div.product-card", timeout=15000)
    html = page.content()
    browser.close()

Then run the same JSON and marker checks on html before you trust it.

FAQ

Does a 200 status code mean the page is complete?

No. The status code is sent before the body. It tells you the server started a successful response, not that the response finished. Completeness has to be checked on the body.

Why does BeautifulSoup show closing tags that were not in the response?

HTML parsers close any open elements when the input ends, the same way browsers do. That makes a truncated document look well-formed after parsing. Check the raw text before you parse it.

Should I retry a truncated response?

Yes, with a limit. Most truncations are transient and the same URL returns the full page on the next attempt. Use a fresh connection, cap the attempts at two or three, and log the reason so you can spot a pattern.

What is the most reliable single check?

Parsing embedded JSON such as __NEXT_DATA__ or JSON-LD. Unlike HTML, JSON has no forgiving parser, so a cut-off block always fails to parse.

Wrapping Up

Truncated responses are rare, which is exactly why they hurt. They slip past status checks and parsers, then show up weeks later as gaps in your data. A guard of a few lines that checks the raw body for a closing tag, valid embedded JSON and an end-of-page marker catches nearly all of them, and a bounded retry recovers the data.

If running browsers and proxies yourself is the part you would rather skip, ScrapeUnblocker fetches and renders pages for you and returns the HTML in one API call. The docs show how to make the request. Keep the completeness guard in front of your parser whatever your data source is: it checks the one thing a status code cannot.