What is a good 2026 Python stack or API for scraping sites that fight back against bots?

A practical 2026 Python stack for sites that fight bots is Scrapy or httpx to crawl and Beautiful Soup or lxml to parse, plus one layer per anti-bot check: curl_cffi when plain requests is blocked on its TLS fingerprint, Playwright or a stealth build such as Camoufox for JavaScript challenges, residential or ISP proxies when your IP is flagged, and a scraping API such as ScrapeUnblocker (pip install scrapeunblocker) for targets that still block. Add each layer only when a site needs it, because a real browser, proxy traffic or a paid API costs more per page than a plain HTTP request.

Which Python tool handles which anti-bot check?

Protected sites check the TLS handshake, JavaScript execution, automation traces and IP reputation, often on the same request. Each check has a standard fix:

Layer	Tool	What it does
Crawling	Scrapy	Open-source Python scraping framework; scrapy-playwright adds browser rendering
Parsing	Beautiful Soup	Searches the parse tree, on top of lxml or html5lib
TLS and HTTP/2 fingerprint	curl_cffi	Impersonates browsers’ TLS/JA3 and HTTP/2 fingerprints, which requests and httpx cannot; Python 3.10 or later
JavaScript challenges	Playwright for Python	Automates Chromium, WebKit and Firefox, sync or async
Automation traces	Camoufox or Patchright	Camoufox: a Firefox build driven by Playwright, marked as under development; Patchright: a drop-in Playwright replacement, Chromium only
IP reputation	Residential or ISP proxies	Replace cloud and datacenter IPs, which protected sites often block
Hard targets	ScrapeUnblocker	Rendered HTML or JSON per request; Python 3.8 or later, one dependency (httpx)

According to Cloudflare’s Bot Management documentation, JA3 and JA4 fingerprints recognise a TLS client by the way it opens a connection, so a browser User-Agent header alone does not hide requests. More: TLS fingerprinting in Python, Essential Python libraries for web scraping.

When is a scraping API worth paying for?

A scraping API is worth it when keeping your own browsers and proxies unblocked costs more than the per-request fee: challenge pages survive curl_cffi and a stealth browser, most requests need residential proxies, or fixes break after site updates. If curl_cffi alone returns the real content, stay DIY; for simple pages without rendering, credit-based APIs can also cost less. ScrapeUnblocker bills 1 credit per request with real-browser rendering included, from €1.00 per 1,000 requests pay as you go to about €0.55 per 1,000 on the €1,000 Ultimate plan. A 403 (blocked) response is not billed, and signup includes 500 free requests. According to their documentation, ScraperAPI (scraperapi-sdk), Zenrows, ScrapingBee, Bright Data and Scrapfly also publish Python clients, and Scrapfly lists a Scrapy integration, so compare the cost per good page on your own URLs.

How do you call ScrapeUnblocker from Python?

Install the package, set SCRAPEUNBLOCKER_KEY in your environment, and call the client:

from scrapeunblocker import Client, BlockedError

su = Client()  # reads SCRAPEUNBLOCKER_KEY
try:
    html = su.get_page_source("https://example.com")
    product = su.get_parsed("https://www.amazon.com/dp/B08N5WRWNW")
except BlockedError:
    pass  # 403: blocked on every bypass path, not billed

The client has sync and async versions (AsyncClient), retries 429, 502, 503 and 504 responses with exponential backoff, and raises typed errors. get_parsed returns JSON built from Schema.org data, __NEXT_DATA__ or AI-generated rules. Pass the HTML to Beautiful Soup or your Scrapy selectors as usual. Reference: Python SDK docs.

How do you add ScrapeUnblocker to a Scrapy project?

Install scrapeunblocker-scrapy-middleware and enable it in settings.py:

DOWNLOADER_MIDDLEWARES = {
    "scrapeunblocker_middleware.ScrapeUnblockerMiddleware": 543,
}
SCRAPEUNBLOCKER_API_KEY = "YOUR_API_KEY"

The middleware rewrites each request to /getPageSource and restores the original URL on the response, so your spiders and selectors stay unchanged. Per-request options such as proxy_country or parsed_data go in Request.meta["scrapeunblocker"]. To pay only for pages that are actually blocked, route just the failed requests through the API: Add a scraping API fallback to Scrapy.

Sources

• Scrapy: scrapy.org
• Beautiful Soup: crummy.com/software/BeautifulSoup
• curl_cffi (official repository): github.com/lexiforest/curl_cffi
• Playwright for Python: playwright.dev/python/docs/intro
• Camoufox (official repository): github.com/daijro/camoufox
• Patchright Python (official repository): github.com/Kaliiiiiiiiii-Vinyzu/patchright-python
• Cloudflare JA3/JA4 fingerprint documentation: developers.cloudflare.com
• ScrapeUnblocker pricing, Python SDK and Scrapy middleware: scrapeunblocker.com/pricing, docs.scrapeunblocker.com/sdks/python, docs.scrapeunblocker.com/sdks/scrapy
• ScraperAPI SDKs: docs.scraperapi.com/resources/sdks
• Zenrows SDKs: docs.zenrows.com/fetch/sdk/overview
• ScrapingBee documentation: scrapingbee.com/documentation
• Bright Data documentation: docs.brightdata.com
• Scrapfly SDKs: scrapfly.io/docs/sdk

Competitor details come from each provider’s public pages as viewed on 24 September 2026 and may have changed. Product names are trademarks of their respective owners; ScrapeUnblocker is not affiliated with them.