Add a Scraping API Fallback to Your Scrapy Project

Your Scrapy spider works. It pulls thousands of pages a day, parses cleanly, and costs you nothing but bandwidth. Then one morning the logs fill with 403 responses, or worse, 200 responses that contain a challenge page instead of the data you wanted. The site added an anti-bot layer, and your direct requests stopped getting through.

The reflex is to route everything through a scraping API. That works, but it is wasteful. Most of your requests were never blocked, and sending all of them through a paid API means paying for pages you could have fetched for free. A better pattern is a fallback: fetch directly first, and only escalate to the API when a request actually fails.

This guide shows you how to build that fallback into Scrapy as a downloader middleware, with proper detection, retry logic, and error handling. The code is complete and you can drop it into an existing project.

Why a Fallback Instead of an All-or-Nothing Switch

A scraping API is a paid resource. Every request through it costs a credit. If 90% of your target pages are unprotected, routing them all through the API multiplies your bill for no benefit.

A fallback flips the logic:

• Direct first. Scrapy sends its normal request. If the site responds with clean data, you are done and you paid nothing.
• Escalate on failure. If the response is blocked, missing, or malformed, the middleware retries the same URL through the API.
• Give up gracefully. After a set number of API attempts, the request is dropped or logged so it does not loop forever.

This keeps your credit spend proportional to how hard the site actually is. It also means you can point a spider at a mixed set of domains, some easy and some protected, without splitting your code into two paths.

Where the Fallback Fits in Scrapy

Scrapy processes every request through a chain of downloader middlewares before and after the actual download. This is the right place for a fallback, because a middleware can:

1. See the response Scrapy got from the direct request.
2. Decide whether that response counts as a block.
3. Replace it with a fresh request routed through the API.

You do not touch your spider’s parsing logic at all. The spider asks for a URL and receives a working response. Whether that response came directly or through the API is invisible to it.

Detecting a Block

The hardest part of a fallback is not the API call. It is deciding when to trigger it. A naive check on response.status == 403 misses the most common modern failure: the soft block, where the server returns 200 OK with a challenge or “please verify you are human” page instead of real content.

Build your detection around a few signals:

• Hard status codes. 403, 429, and 503 are explicit blocks or rate limits.
• Suspicious body on a 200. Very short bodies, or bodies containing known challenge markers, are soft blocks.
• Missing expected content. If the page should contain a product grid and the selector returns nothing, treat it as a failed fetch.

Here is a detection helper you can tune per project:

BLOCK_STATUS = {403, 429, 503}
CHALLENGE_MARKERS = (
    b"captcha",
    b"cf-challenge",
    b"just a moment",
    b"verify you are human",
)

def looks_blocked(response):
    if response.status in BLOCK_STATUS:
        return True
    body = response.body[:20000].lower()
    if len(response.body) < 500:
        return True
    return any(marker in body for marker in CHALLENGE_MARKERS)

Keep the marker list short and specific. If you match too broadly, you will send clean pages through the API and burn credits. If you want a deeper look at why a 200 can still be a block, see our guide on detecting soft blocks in web scraping.

The Fallback Middleware

Now the middleware itself. It watches every response, and when looks_blocked returns True, it rebuilds the request to go through the scraping API instead of the origin. It tracks how many times it has retried a given URL through a request meta flag so it never loops.

ScrapeUnblocker’s fetch endpoint takes the target URL as a query parameter and an API key in a header, so the fallback is a matter of rewriting the request URL and swapping the headers.

import logging
from urllib.parse import quote, urlencode

from scrapy.exceptions import IgnoreRequest
from scrapy.http import Request

logger = logging.getLogger(__name__)

API_ENDPOINT = "https://api.scrapeunblocker.com/getPageSource"


class ScrapingApiFallbackMiddleware:
    def __init__(self, api_key, max_api_retries):
        if not api_key:
            raise ValueError("SCRAPEUNBLOCKER_API_KEY is not set")
        self.api_key = api_key
        self.max_api_retries = max_api_retries

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            api_key=crawler.settings.get("SCRAPEUNBLOCKER_API_KEY"),
            max_api_retries=crawler.settings.getint("API_FALLBACK_MAX_RETRIES", 2),
        )

    def _build_api_request(self, original, attempt):
        params = {"url": original.meta.get("origin_url", original.url)}
        # Add extra options here if a site needs them, for example:
        # params["proxy_country"] = "us"
        api_url = f"{API_ENDPOINT}?{urlencode(params, quote_via=quote)}"

        return original.replace(
            url=api_url,
            method="POST",
            headers={"x-scrapeunblocker-key": self.api_key},
            meta={
                **original.meta,
                "origin_url": original.meta.get("origin_url", original.url),
                "api_attempt": attempt,
                "download_slot": "scrapeunblocker-api",
            },
            dont_filter=True,
        )

    def process_response(self, request, response, spider):
        attempt = request.meta.get("api_attempt", 0)

        if not looks_blocked(response):
            return response

        if attempt >= self.max_api_retries:
            logger.warning(
                "Giving up on %s after %d API attempts",
                request.meta.get("origin_url", request.url),
                attempt,
            )
            raise IgnoreRequest(f"Blocked after {attempt} API retries")

        next_attempt = attempt + 1
        logger.info(
            "Blocked, routing through API (attempt %d): %s",
            next_attempt,
            request.meta.get("origin_url", request.url),
        )
        return self._build_api_request(request, next_attempt)

A few details worth understanding:

• origin_url in meta. Once a request is rewritten to point at the API, request.url is the API URL, not the page you wanted. Storing the original target in meta["origin_url"] lets every retry rebuild the API call from the real URL.
• api_attempt counter. Each escalation increments this. When it reaches max_api_retries, the middleware raises IgnoreRequest and stops.
• download_slot. Setting a shared slot for API requests lets you throttle them independently from your direct traffic (more on that below).
• dont_filter=True. Without this, Scrapy’s dupe filter would drop the retry because it targets a URL the spider already visited.

Wiring It Into Settings

Enable the middleware and set your key and limits in settings.py:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ScrapingApiFallbackMiddleware": 610,
}

SCRAPEUNBLOCKER_API_KEY = "YOUR_API_KEY"
API_FALLBACK_MAX_RETRIES = 2

# Throttle API traffic separately from direct requests.
DOWNLOAD_SLOTS = {
    "scrapeunblocker-api": {"concurrency": 4, "delay": 0},
}

The priority number 610 places the middleware just after Scrapy’s built-in RetryMiddleware (which sits at 550) so that ordinary transient errors get retried directly first, and only persistent blocks fall through to the API. Read your key from an environment variable rather than hardcoding it:

import os
SCRAPEUNBLOCKER_API_KEY = os.environ["SCRAPEUNBLOCKER_API_KEY"]

Handling Errors and Timeouts

Blocks are not the only failure mode. The API request itself can time out or return an error, and you want that handled without crashing the crawl. Add a process_exception hook to the same middleware so network errors on a direct request also trigger the fallback:

    def process_exception(self, request, exception, spider):
        attempt = request.meta.get("api_attempt", 0)
        if attempt >= self.max_api_retries:
            return None  # let Scrapy handle the failure normally

        logger.info(
            "Download error (%s), routing through API: %s",
            type(exception).__name__,
            request.meta.get("origin_url", request.url),
        )
        return self._build_api_request(request, attempt + 1)

Combine this with Scrapy’s own retry and timeout settings so a slow API call does not stall the whole crawl:

DOWNLOAD_TIMEOUT = 60
RETRY_ENABLED = True
RETRY_TIMES = 2

Because the API renders protected pages with a real browser, its responses are slower than a raw fetch. A timeout of 60 seconds gives it room to work without hanging your spider indefinitely.

Testing the Fallback

Before running at scale, confirm the two paths behave:

• Point the spider at an unprotected page and check the logs show no API attempts. That proves you are not wasting credits on easy pages.
• Point it at a known protected page and confirm you see the “routing through API” log line, followed by a successful parse. That proves escalation works.
• Force a failure by setting max_api_retries to 0 and confirm the request is dropped cleanly with the “giving up” warning rather than looping.

FAQ

Does the fallback slow down my crawl? Only for pages that get blocked. Direct requests run at full speed. Escalated requests are slower because the API renders them with a real browser, which is the tradeoff for getting past the block at all. The separate download slot keeps that slower traffic from throttling your direct requests.

How do I avoid paying for pages that were not really blocked? Tune looks_blocked. Keep the challenge marker list specific to what your targets actually return, and log every escalation during testing so you can spot false positives before they cost you.

Can I use this with Scrapy’s AutoThrottle? Yes. AutoThrottle adjusts delays based on response latency. Putting API requests in their own download slot keeps their higher latency from dragging down the delay calculation for your direct traffic.

What if a page needs JavaScript rendering or a specific country? Add the relevant parameters to the params dict in _build_api_request. For example, set a proxy_country value to route through a given region. The rest of the middleware stays the same.

Wrapping Up

A scraping API fallback gives you the best of both models: free, fast direct fetches for the bulk of your pages, and a reliable escalation path for the handful of domains that fight back. The whole thing lives in one downloader middleware, so your spiders stay clean and your credit spend stays proportional to how hard each site actually is.

If you want to try the escalation path, ScrapeUnblocker handles the anti-bot bypass and browser rendering behind a single endpoint, billed at one credit per request with JavaScript rendering included. Drop the middleware above into your project, point SCRAPEUNBLOCKER_API_KEY at your key, and your existing spiders keep running while the hard pages quietly start working again.