Retry Logic for Web Scrapers: 6 Mistakes That Turn Errors Into Blocks

A few hundred pages into a run, your scraper starts receiving 429 Too Many Requests. The retry wrapper sends those requests again. Soon, most responses are 403 or challenge pages.

Your parser has not changed. Your traffic has: retries added requests just as the site started rejecting them. A retry policy that treats every failure as a temporary server problem can make a rate limit or anti-bot block worse.

The fix starts with classifying the response. Then decide whether to wait, change the client identity, or stop. These six mistakes undermine that decision.

1. Retrying Immediately

This is the retry loop most scrapers start with:

for attempt in range(3):
    response = session.get(url)
    if response.status_code == 200:
        break

One rejected request becomes three requests in quick succession. If the response was a rate limit, the loop increases traffic at exactly the wrong time.

Library defaults do not necessarily add a delay:

• urllib3.util.Retry, which provides retry handling for requests adapters, defaults to backoff_factor=0. It adds no backoff unless you configure one.
• Scrapy’s RetryMiddleware retries 408, 429, 500, 502, 503, 504, 522, and 524 up to RETRY_TIMES = 2 additional times. It has no retry-delay setting. With the default RETRY_PRIORITY_ADJUST = -1, the retry enters the scheduler at a lower priority and can run as soon as a download slot is available.

Use exponential backoff with jitter. Backoff increases the delay after repeated failures. Jitter randomizes it so workers that fail together do not all retry together.

The full-jitter approach described in the AWS Architecture Blog post Exponential Backoff And Jitter chooses a random delay between zero and min(cap, base * 2 ** attempt).

This configuration uses requests with urllib3 2.x to add backoff and jitter at the HTTP adapter layer. Use it only if that layer owns your retries.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(
    total=3,
    backoff_factor=1,          # 1 s, 2 s, 4 s ... capped by backoff_max (120 s)
    backoff_jitter=1.0,        # adds 0-1 s of random spread
    status_forcelist=[429, 500, 502, 503, 504],
    respect_retry_after_header=True,
)

session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))

urllib3’s backoff_jitter adds a random amount to its calculated backoff. That is additive jitter, not full jitter: workers still share the underlying backoff curve. With tenacity, wait_random_exponential(multiplier=1, max=60) provides full jitter directly.

When retries run out on a status in status_forcelist, requests raises requests.exceptions.RetryError instead of returning the final response. Outer layers should record that failure, not start another retry loop.

2. Ignoring Retry-After

Before calculating your own delay, inspect Retry-After. A 429 or 503 can use it to tell you when to try again.

Per MDN, the header accepts either a delay in seconds, such as Retry-After: 120, or an HTTP date. Your parser needs to handle both forms.

If the server asks for 120 seconds and your backoff picks two, you are still inside the window. Some rate limiters count rejected requests too, so early attempts can keep you in it.

The retry behavior differs by library:

• urllib3: respect_retry_after_header is enabled by default. Its special handling covers 413, 429, and 503; these responses can qualify for a retry with Retry-After even without an entry in status_forcelist, provided the other retry conditions allow it. Versions exposing retry_after_max cap this wait at six hours by default.
• tenacity: It does not interpret HTTP responses. Your wait function must read and parse the header.
• httpx: HTTPTransport(retries=3) retries ConnectError and ConnectTimeout, not HTTP status responses. A 429 returns to your application code.

This tenacity wait function uses the server’s delay when there is one and falls back to full jitter otherwise:

import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

import httpx
from tenacity import (retry, retry_if_exception_type,
                      stop_after_attempt, wait_random_exponential)


class Retryable(Exception):
    def __init__(self, response):
        super().__init__(response.status_code)
        self.response = response


def retry_after_seconds(response):
    value = response.headers.get("Retry-After", "").strip()
    if not value:
        return None
    if value.isdigit():
        return int(value)
    try:
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    if when.tzinfo is None:
        when = when.replace(tzinfo=timezone.utc)
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())


fallback = wait_random_exponential(multiplier=1, max=60)


def wait_for_server(retry_state):
    exc = retry_state.outcome.exception()
    if isinstance(exc, Retryable):
        seconds = retry_after_seconds(exc.response)
        if seconds is not None:
            return min(seconds, 300) + random.uniform(0, 2)
    return fallback(retry_state)


@retry(
    retry=retry_if_exception_type((Retryable, httpx.TransportError)),
    wait=wait_for_server,
    stop=stop_after_attempt(3),
    reraise=True,
)
def fetch(client: httpx.Client, url: str) -> httpx.Response:
    response = client.get(url)
    if response.status_code in (429, 502, 503, 504):
        raise Retryable(response)
    response.raise_for_status()
    return response

Do not extend this status-based policy to every 403. A challenge response needs classification before another attempt, as covered in mistake 5.

3. Retrying Responses That Are Not Failures

A short response is not necessarily a failed response. Paginated JSON or AJAX endpoints can return an empty list, an empty body, or a small HTML fragment after the last page. A fragment may contain a doctype and a few <li> elements without a full <html> or <head> wrapper.

If your validator treats all of those shapes as broken, retries will keep fetching the same valid response.

Validate against what the endpoint is expected to return, not a minimum body length. If two different client identities return the same short response, it is probably the real answer. Check the pagination state before retrying.

The inverse error is accepting a challenge page because its status is 200. Status-only validation lets that HTML enter your dataset instead of triggering block handling.

Check both the status and the content you need to extract. See HTTP 200 But Blocked: Detecting Soft Blocks for challenge indicators and Truncated HTML Responses for incomplete bodies.

4. Pinning the Proxy Country on Every Attempt

Country targeting restricts which proxy exits your scraper can use. A provider may have a large residential pool overall but relatively few exits in the country you select.

If every request and retry must use that subset, requests concentrate on fewer IP addresses. Per-IP traffic rises, and the site may respond with blocks or slider CAPTCHAs. Rotating the proxy while retaining the same country constraint does not necessarily give you a substantially different pool.

First, establish whether the country restriction is necessary:

• Fetch the same page through two countries.
• Compare the fields your scraper actually uses, such as price, currency, stock, and language.
• If the relevant fields match, remove the country restriction.
• If they differ, keep the required country. On retry, wait and lower concurrency instead of drawing another exit from the same small pool.

A nearby market is an alternative only after you have verified that it serves the same relevant version of the page. Do not trade correct data for an easier request.

5. Retrying a Block With the Same Identity

A 403 alone does not tell you why access failed: it can be a login requirement, a site rule or an anti-bot decision. When Cloudflare, DataDome or Akamai returns a challenge, the decision involves your IP address, TLS fingerprint, headers and session cookies. Repeating the same client state sends the same signals again.

Cloudflare provides an explicit marker for its challenge responses: cf-mitigated: challenge. Inspect that header as well as the body. Other challenge responses require their own content checks.

Use response classification to choose the next action:

Response	Likely Interpretation	Before You Retry
429 with Retry-After	Rate limit	Wait the stated time and reduce concurrency for that host
502, 503, 504, 522, or 524 without challenge markup	Origin or edge failure	Back off; retaining the same identity may be appropriate
403 with cf-mitigated: challenge or recognizable challenge HTML	Anti-bot challenge	Use a new IP and fresh session cookies, or a different client
200 containing CAPTCHA or “verify you are human” HTML	Soft block	Handle it as a challenge, not a successful page
Connection reset during the TLS handshake	Network failure or possible edge blocking	Allow one retry on a new connection and proxy
404 or 410	Missing or removed page	Do not retry as a transient failure

For a confirmed block, escalate from cheaper to more expensive changes: a new proxy session, then a different client fingerprint, then a browser with Playwright. For example, curl_cffi exposes an impersonate target for selecting a client fingerprint. Change it deliberately rather than replaying the same blocked configuration.

Each escalation adds time and cost. Move to the next step only after the previous one fails, and keep the entire ladder inside one attempt budget.

Also isolate repeatedly failing URLs. If one URL is blocked across two or three identities while other pages on the host load successfully, stop spending working identities on it. A stricter path rule or a trap may be involved. Park the URL for inspection.

6. Letting Every Layer Retry

Independent retry layers multiply attempts. They do not merely add them.

Suppose your HTTP client retries three times, giving four attempts. A Scrapy middleware or custom wrapper adds two retries, giving three executions of that client operation. Then the task queue runs the failed task again; Celery’s autoretry_for uses a default max_retries of three, giving four task runs.

The total is 4 × 3 × 4 = 48 requests for one persistently failing URL. When many URLs fail together, all those layers amplify traffic while the site is already rejecting requests.

The Google SRE book’s chapter on handling overload describes this same retry amplification. Its controls apply directly to scraper design:

• Choose one retry owner. Other layers return failures to it instead of starting their own loops.
• Cap attempts per request. The book describes allowing three failures before returning the failure to the caller.
• Apply a retry budget. Its example permits retries only while they remain below 10% of all requests.

For a scraper, track the budget per host. Add a circuit breaker that temporarily stops requests to a host when the block rate crosses your threshold. Without it, a bounded retry loop can still overwhelm a host when enough workers run it at once. The budget limits extra traffic; the breaker pauses a host that is rejecting too many requests:

import random
import time
from collections import defaultdict, deque
from urllib.parse import urlsplit


class HostGuard:
    """Per-host retry budget plus a circuit breaker."""

    def __init__(self, retry_ratio=0.1, window=200,
                 block_threshold=0.3, cooldown=600):
        self.retry_ratio = retry_ratio
        self.block_threshold = block_threshold
        self.cooldown = cooldown
        self.sent = defaultdict(lambda: deque(maxlen=window))     # True = retry
        self.blocked = defaultdict(lambda: deque(maxlen=window))  # True = block
        self.open_until = {}

    def can_send(self, host):
        return time.monotonic() >= self.open_until.get(host, 0)

    def can_retry(self, host):
        sent = self.sent[host]
        budget = self.retry_ratio * max(len(sent), 10)
        return self.can_send(host) and sum(sent) < budget

    def record(self, host, is_retry, blocked):
        self.sent[host].append(is_retry)
        recent = self.blocked[host]
        recent.append(blocked)
        if len(recent) >= 50 and sum(recent) / len(recent) >= self.block_threshold:
            self.open_until[host] = time.monotonic() + self.cooldown
            recent.clear()


def crawl(urls, fetch_once, max_attempts=3):
    """fetch_once(url, attempt) returns an object with .ok, .blocked, .retryable."""
    guard = HostGuard()
    parked = []
    for url in urls:
        host = urlsplit(url).hostname
        if not guard.can_send(host):
            parked.append(url)
            continue
        result = fetch_once(url, attempt=0)
        guard.record(host, is_retry=False, blocked=result.blocked)
        attempt = 1
        while (not result.ok and result.retryable
               and attempt < max_attempts and guard.can_retry(host)):
            time.sleep(random.uniform(0, min(60, 2 ** attempt)))
            result = fetch_once(url, attempt=attempt)
            guard.record(host, is_retry=True, blocked=result.blocked)
            attempt += 1
        if not result.ok:
            parked.append(url)
    return parked

fetch_once must perform the response classification from mistakes 3 and 5. If another attempt is allowed after a block, it must arrange the identity change before that attempt runs.

The example loop is sequential. With asyncio or a thread pool, share one HostGuard across workers targeting the same host. Separate guards would give each worker its own budget and hide the aggregate failure rate.

The trade-off: these controls leave some URLs unfinished on purpose. You give up completeness of one run so you do not lose the host for the next ten. Park the rest for a later run at a lower rate.

A breaker also cannot tell an outage from a block on your identity; both trip it. Inspect saved responses before deciding whether to change your client or simply wait.

FAQ

How many times should a scraper retry a failed request?

Two or three total attempts per URL, owned by one layer, is enough for transient failures such as timeouts and 503 responses. If it still fails, stop the current loop, park the URL and try it in a later run.

Should I retry a 403 Forbidden response?

Not blindly with the same request. First distinguish an anti-bot response from an access restriction. For an anti-bot block, change the relevant IP, fingerprint, or session state before another attempt. Changing identity will not supply a required login or permission.

What is the difference between exponential backoff and jitter?

Exponential backoff increases the delay between attempts, for example 1, 2, 4, and 8 seconds. Jitter randomizes the delay so workers that fail together do not retry together. Use both, while honoring Retry-After when the server supplies it.

Does Scrapy wait between retries?

No. RetryMiddleware reschedules the request with lower priority and no delay. Turn on AutoThrottle (AUTOTHROTTLE_ENABLED = True), which never lets a non-200 response lower the delay, keep RETRY_TIMES low, and write your own middleware for real per-request backoff.

Count Your Retry Layers First

Open your scraper and list every retry source: the HTTP adapter, Scrapy settings, your wrapper, the task queue, and any scheduled rerun of failed batches. Multiply the attempts. If the result exceeds three, choose one owner and disable the other retry loops before tuning delays.

If you would rather not maintain the proxy, fingerprint, and browser changes between attempts, ScrapeUnblocker handles proxies, rendering, and blocks on its side. Its errors documentation states that a 403 block, timeout, or 429 is not billed. You can keep your application-side retry policy in one place, around that API call.