HTTP 200 but Blocked: How to Detect Soft Blocks in Web Scraping

Your scraper finishes a run. Every request returned HTTP 200. No exceptions, no retries, no errors in the log. Then you open the output and half the records are missing, prices are blank, and a few rows contain the phrase “please verify you are human.” The status code lied to you.

This is one of the most expensive failure modes in web scraping, because it is silent. A hard block (403, 429, a connection reset) is loud - you see it, you handle it. A soft block returns 200 with a body that is not the page you asked for, so your parser runs, finds nothing, and moves on. The scrape “succeeds” while your dataset quietly rots.

This guide covers what soft blocks look like from your side, why sites serve them, and how to detect them reliably with open-source tools before bad data reaches your database.

What a Soft Block Actually Is

A soft block is any response that carries a success status code but does not contain the content you requested. The server decided you look like a bot, but instead of rejecting you outright, it hands back something else and marks it 200 OK. Common variants:

• A challenge or interstitial page. The body is a JavaScript challenge, a “checking your browser” holding page, or a CAPTCHA wall. Status: 200. Content: not your data.
• A consent or region wall. A cookie or GDPR consent screen renders instead of the article. The real content sits behind an interaction you never performed.
• A stripped or partial page. You get the shell of the page - header, footer, layout - but the product grid, listing table, or article body is empty or truncated. Sometimes you get 3 items when the page really has 60.
• A generic “something went wrong” page. A friendly error page served with 200 so that monitoring tools do not flag it.
• A honeypot or decoy response. Fake or randomized data designed to look plausible while being useless.

The common thread: the HTTP layer says everything is fine, and only the content tells the truth.

Why Sites Serve 200 Instead of 403

It feels backwards. If a site does not want to serve you, why not just say no? Three practical reasons:

1. It breaks naive scrapers quietly. Many scrapers only check response.status_code. Serving a 200 with junk means those scrapers never trigger their retry or alerting logic. Your pipeline degrades instead of failing, which is harder to notice and harder to debug.
2. It protects real users. Anti-bot systems are not perfect and sometimes flag genuine visitors. A soft challenge (solve this, click here) is a gentler fallback than a hard 403 that would lock a real customer out.
3. It muddies your signal. If blocks always came back as 403, you could measure your block rate precisely and adapt. Mixing blocks into 200s makes your success metrics look healthier than they are, which slows down your response.

Understanding the motive matters because it tells you where to look: never trust the status code alone, and always validate the body.

How to Detect Soft Blocks

Detection comes down to one principle - validate the content, not just the transport. Here is a layered approach, cheapest checks first.

1. Check the Response Size

The fastest signal. A real product page might be 200-500 KB of HTML. A challenge page or stripped shell is often a fraction of that. Record a baseline for each page type and flag anything that falls well below it.

import requests

resp = requests.get(url, headers=headers, timeout=20)
size = len(resp.content)

if size < 15_000:  # tune per site
    raise ValueError(f"Suspiciously small response: {size} bytes")

Size alone produces false positives, so treat it as a first filter, not a verdict.

2. Scan for Block Signatures

Look for phrases and markers that only appear on challenge or error pages. Keep a per-site list, because the wording varies.

BLOCK_MARKERS = [
    "verify you are human",
    "checking your browser",
    "enable javascript and cookies",
    "access denied",
    "unusual traffic",
    "captcha",
    "cf-challenge",
    "px-captcha",
]

body = resp.text.lower()
if any(marker in body for marker in BLOCK_MARKERS):
    raise ValueError("Block signature found in body")

This catches the majority of interstitial and challenge pages. Update the list whenever you see a new wall.

3. Assert on the Content You Expect

The most reliable check is positive, not negative. Do not just look for signs of a block - confirm the thing you came for is actually present. If you are scraping a product page, the price selector must resolve. If you are scraping a listing page, the item count must be within a sane range.

from parsel import Selector

sel = Selector(resp.text)
items = sel.css("div.product-card")

if len(items) < 5:
    raise ValueError(f"Only {len(items)} items - expected a full page")

price = sel.css("span.price::text").get()
if not price:
    raise ValueError("Price selector empty - page likely incomplete")

This is what catches the nastiest variant: the partial page. A stripped response has no block markers and can be a normal size, but the data simply is not there. Only a positive assertion catches it.

4. Detect Partial and Inconsistent Pages

Some sites return the full page most of the time and a truncated version at random, especially under load or when they suspect automation. The same URL gives you 60 items on one request and 3 on the next, both with status 200.

Defend against it with a completeness heuristic and a bounded retry:

def fetch_complete(url, min_items=20, attempts=3):
    best = None
    for _ in range(attempts):
        resp = requests.get(url, headers=headers, timeout=20)
        sel = Selector(resp.text)
        items = sel.css("div.product-card")
        if best is None or len(items) > len(best):
            best = items
        if len(items) >= min_items:
            return items  # good enough, stop early
    return best  # return the fullest render we saw

Keeping the fullest of several attempts, rather than the first, turns flaky partial responses into usable data.

5. Compare Against a Known-Good Baseline

For high-value pages, store a snapshot of a healthy response - its size, its item count, the presence of key selectors. On each run, compare against the baseline and alert when the shape drifts. A sudden 80% drop in item count across many pages is a far better block signal than any single request in isolation.

Turn Detection Into a Reusable Guard

Do not scatter these checks through your spiders. Wrap them in one validation function that every response passes through before parsing. Fail loudly - raise, log with the URL, and route to a retry queue - so a soft block becomes a visible event instead of silent data loss.

def validate(resp, min_items=5):
    if len(resp.content) < 15_000:
        raise ValueError("response too small")
    body = resp.text.lower()
    if any(m in body for m in BLOCK_MARKERS):
        raise ValueError("block signature")
    sel = Selector(resp.text)
    if len(sel.css("div.product-card")) < min_items:
        raise ValueError("content missing")
    return sel

One choke point means one place to tune thresholds and add new signatures as sites change.

When Detection Is Not Enough

Detection tells you a request failed. It does not fix the underlying reason you were flagged. If your validated block rate climbs steadily - say from 1% to 8% over a week across many targets - that is usually the site tightening its anti-bot posture, not a bug in your parser. At that point the reader-side levers are the usual ones: rotate IPs, use realistic browser fingerprints with tools like curl_cffi or an undetected browser, add human-like delays, and render JavaScript with Playwright or Selenium when the content requires it.

At some scale, maintaining all of that yourself becomes its own project. A managed unblocking service such as ScrapeUnblocker handles the fingerprinting, rotation, and rendering behind a single API call, so you get the real page back and can focus your validation logic on data quality rather than block avoidance. You can read more about how it fits into a scraping stack in the developer docs.

FAQ

Does a 200 status code mean my scrape worked? No. A 200 only means the server responded with something. It says nothing about whether that something is the content you wanted. Always validate the body.

How do I tell a soft block from a genuinely empty page? Use a positive assertion tied to the page type and a baseline. If a category page normally has 50 items and returns 0, that is a block or a parser break, not an empty category. Cross-check with response size and block signatures.

Why do I get different results for the same URL? Some sites serve partial or randomized responses under load or when they suspect automation. Fetch a few times and keep the most complete render, and log the variance so you can spot when it gets worse.

Should I retry a soft block immediately? A bounded retry (2-3 attempts) helps with flaky partial pages. If the block persists, retrying the same way just wastes requests - change something first, such as your IP, headers, or rendering approach.

Key Takeaways

• Status codes describe the transport, not the content. Never trust 200 alone.
• Layer your checks: response size, block signatures, and - most importantly - positive assertions on the data you expect.
• Partial pages are the sneakiest soft block. Only a content-completeness check catches them.
• Centralize validation into one guard so every response is verified before it reaches your parser.
• A rising validated block rate is a signal to change your approach, not just retry harder.

Treat every response as guilty until proven complete, and soft blocks stop being silent.