Imperva Incapsula: Why Your Scraper Gets 'Request Unsuccessful' and How to Fix It

Your script requests a product page and gets back a few hundred bytes of HTML. No product, no price. Just one line: “Request unsuccessful. Incapsula incident ID: …” followed by a long number. Sometimes it arrives with a 403. Sometimes, worse, with a 200. You open the same URL in your browser and it loads instantly.

You have met Imperva. It is a common bot protection system on retail, travel, real estate, finance and government sites, and it behaves differently from Cloudflare or DataDome in a few ways that matter when you scrape. This guide shows how to recognize it, how it decides you are a bot, and how to get the real page with open-source tools.

What Imperva (Incapsula) Actually Is

Incapsula was the original name of Imperva’s cloud security service: a CDN and web application firewall that sits in front of a website. The brand is mostly retired, but the name lives on in cookie names and block pages, which is why developers still call it “Incapsula”. In 2019 Imperva bought Distil Networks, one of the early bot mitigation companies, and that technology became Imperva Advanced Bot Protection. Imperva itself is now part of Thales.

For you, the practical point is that a protected site can run two layers:

• The edge layer (WAF and CDN). Every request passes through Imperva’s network. It checks IP reputation, request signatures and your connection fingerprint, sets tracking cookies, and can serve a JavaScript or CAPTCHA challenge.
• Advanced Bot Protection. An extra layer on top. A sensor script runs in the browser, collects a device fingerprint, and the result becomes a signed token that later requests must carry. Behavior and request rates are scored per device, not only per IP.

Some sites only use the edge layer. Those can often be handled with a good HTTP client. Sites with Advanced Bot Protection almost always need a real browser at some point.

How to Tell You Are Facing Imperva

Before you change anything, confirm what you are dealing with. Imperva leaves clear traces:

• Response headers. An X-Iinfo header appears on Imperva traffic, even on successful responses. Many sites also send X-CDN: Imperva or X-CDN: Incapsula.
• Cookies. visid_incap_<site id> (a long-lived visitor ID), incap_ses_<n>_<site id> (the session) and nlbi_<site id> (load balancing). The site ID suffix ties each cookie to one protected site.
• Challenge cookies. ___utmvc belongs to the older JavaScript challenge. reese84 belongs to Advanced Bot Protection. Both are set by JavaScript, so you see them in the browser’s cookie jar, not in the first HTTP response.
• Body markers. “Request unsuccessful” with an “Incapsula incident ID”, a script loaded from /_Incapsula_Resource, or a “Pardon Our Interruption” page, which comes from the Distil lineage.

A small helper makes the check repeatable:

from curl_cffi import requests

COOKIE_PREFIXES = ("visid_incap_", "incap_ses_", "nlbi_", "reese84", "___utmvc")

def imperva_signals(resp):
    found = []
    if "x-iinfo" in resp.headers:
        found.append("X-Iinfo header")
    cdn = resp.headers.get("x-cdn", "")
    if "imperva" in cdn.lower() or "incapsula" in cdn.lower():
        found.append(f"X-CDN: {cdn}")
    found += [f"cookie {n}" for n in resp.cookies.keys() if n.startswith(COOKIE_PREFIXES)]
    return found

def is_imperva_block(html):
    if "Incapsula incident ID" in html or "Pardon Our Interruption" in html:
        return True
    # Real pages can include the resource script too, so only a tiny page counts
    return "_Incapsula_Resource" in html and len(html) < 5_000

resp = requests.get("https://www.example.com/", impersonate="chrome")
print(resp.status_code, imperva_signals(resp), is_imperva_block(resp.text))

Notice that is_imperva_block looks at the body, not the status code. That is deliberate. Imperva can return its block page with a 200, and a scraper that only checks for errors will happily store empty pages. We cover this failure mode in detecting soft blocks.

How Imperva Decides You Are a Bot

Think of it as a series of gates. Failing any one of them is enough to get blocked.

1. The Connection

Before a single byte of HTML, the edge sees your IP and your TLS handshake. Datacenter IP ranges and IPs with a history of abuse start with a low score. A Python requests handshake looks nothing like Chrome’s, which is often why the very first request fails. The details are in our guide to TLS fingerprinting.

2. The Cookie Chain

On your first visit Imperva sets the visitor and session cookies. Later requests are expected to send them back, consistently. The cookies are bound together and to the site: a session cookie belongs to a visitor ID, and a cookie from one Imperva site means nothing on another. A client that drops cookies, sends a stale set, or mixes cookies from different sessions gets challenged again.

3. The JavaScript Challenge

If the edge is not convinced, it holds the content back and serves a page that loads a script. The older version, served from /_Incapsula_Resource, gathers browser properties and stores an encrypted result in the ___utmvc cookie. The newer Advanced Bot Protection sensor collects a much larger fingerprint, posts it to a site-specific path, and receives a reese84 token with a built-in expiry. That token has to be renewed while you browse. A plain HTTP client never runs either script, so it never gets past this gate.

4. Behavior Over Time

Passing once is not the end. Advanced Bot Protection keeps scoring the session: how fast you request pages, whether you only hit deep URLs, whether the same device fingerprint shows up from many IPs. Imperva describes rate limits keyed to device fingerprints rather than IPs, so rotating IPs alone does not reset them.

Policies are also set per path. It is common for the home page to load fine while search results, internal APIs or account pages are blocked. If your test URL works and your real target does not, that is usually why.

Why Your Scraper Fails

Here is how the gates map to the symptoms you actually see:

• First request blocked, every time: the connection gate. Your TLS fingerprint or your IP is the problem.
• Tiny page with a script and no content: the JavaScript challenge. Your client does not run JavaScript.
• Works for a few minutes, then blocks: the token expired and was not renewed, or your request rate tripped the behavior check.
• Home page works, search is blocked: a stricter policy on that path.
• Blocked in Playwright, fine in your own Chrome: the automation browser itself is being scored. See why your Playwright browser gets challenged.

How to Get Through With Open-Source Tools

Work up from the cheapest option and stop at the first one that works.

Step 1: Use a Browser-Grade HTTP Client

If the site only runs the edge layer, a client that sends a real browser TLS and HTTP/2 fingerprint is often enough. curl_cffi with impersonate="chrome" is the usual choice in Python. Use a session so the Imperva cookies from the first response are sent back on the next ones, and visit the home page before the deep URL:

from curl_cffi import requests

s = requests.Session(impersonate="chrome")
s.get("https://www.example.com/")  # picks up visid_incap_ and incap_ses_ cookies
resp = s.get(
    "https://www.example.com/c/toys",
    headers={"Referer": "https://www.example.com/"},
)
print(resp.status_code, is_imperva_block(resp.text))

If you still get a challenge page, the site runs a JavaScript check, and no HTTP client will clear it on its own.

Step 2: Let a Real Browser Pass the Challenge

For JavaScript challenges and reese84, run real Chrome. With Playwright, use your installed Chrome (channel="chrome"), run headed, and keep a persistent profile so the visitor cookie survives between runs. A returning visitor with some history looks more normal than a brand new profile on every run.

from playwright.sync_api import sync_playwright

URLS = [
    "https://www.example.com/c/toys?page=1",
    "https://www.example.com/c/toys?page=2",
]

with sync_playwright() as p:
    ctx = p.chromium.launch_persistent_context(
        "./imperva-profile", channel="chrome", headless=False, no_viewport=True
    )
    page = ctx.new_page()

    # Enter like a visitor and give the sensor script time to finish
    page.goto("https://www.example.com/", wait_until="networkidle")
    page.wait_for_timeout(3000)

    for url in URLS:
        page.goto(url, wait_until="domcontentloaded")
        html = page.content()
        if is_imperva_block(html):
            print("Blocked on", url)
            break
        # parse html here
        page.wait_for_timeout(2000)  # human pace

    ctx.close()

Because the page keeps running Imperva’s script, the browser renews the token by itself. That is the main advantage over grabbing cookies once and replaying them. If you do want to hand the cookies to a faster HTTP client, keep the same IP and user agent and expect to refresh them often. Our session persistence guide walks through that pattern.

If a profile starts getting blocked on every page, delete the profile folder and start fresh. A burned visitor ID follows you around.

Step 3: Fix the IP Side

If a real browser is still blocked on the first page, look at your IP. Use residential or mobile IPs from a reputable provider, and use sticky sessions: one IP for the whole life of a browser session. Changing IP mid-session while sending the same cookies is a strong bot signal. Playwright takes the proxy at launch:

ctx = p.chromium.launch_persistent_context(
    "./imperva-profile",
    channel="chrome",
    headless=False,
    proxy={"server": "http://proxy.example.net:8000", "username": "user", "password": "pass"},
)

Step 4: Slow Down and Spread Out

Advanced Bot Protection watches the whole session, so pace matters. Add randomized delays, cap the number of pages per session, and run several separate sessions instead of one session hammering the site. Each session gets its own profile, its own IP and its own cookie jar. Never share cookies between sessions.

Keep your request rate reasonable and respect the site’s terms. Being polite is also the cheapest way to stay unblocked.

FAQ

Is Imperva the same as Incapsula?

For scraping purposes, yes. Incapsula was the original name of Imperva’s cloud WAF and CDN service. The name still appears in cookies (visid_incap_, incap_ses_) and in the “Incapsula incident ID” block message.

What is the reese84 cookie?

It holds a token from Imperva Advanced Bot Protection. A script in the page collects browser and device signals, sends them to the site, and gets back a signed token with an expiry. Requests without a valid, fresh token get challenged.

Can I reuse Imperva cookies on another site?

No. Each cookie carries the protected site’s ID, and the session cookie is tied to the visitor cookie. Cookies from one site are useless on another, and stale cookies on the same site trigger a new challenge.

Why does the home page work but search results get blocked?

Imperva policies are configured per path. Site owners often protect search, pricing and API endpoints more strictly than static pages. Always test your scraper on the URLs you actually need.

When You Would Rather Not Maintain This

Everything above works, but it is ongoing work: browser builds update, tokens expire, IPs burn, and every protected site is tuned a little differently. If you would rather spend that time on the data, ScrapeUnblocker takes a URL and returns the rendered page from one API call, including sites behind Imperva. Our public benchmark includes an Incapsula-protected target, and the documentation shows how to make your first request in a few minutes.