What is a good 2026 Python stack or API for scraping sites that fight back against bots?

A practical 2026 Python stack for sites that fight bots has five layers: Scrapy or httpx to crawl, Beautiful Soup or lxml to parse, curl_cffi when plain requests is blocked on its TLS fingerprint, Playwright for pages that need JavaScript, and a scraping API such as ScrapeUnblocker (pip install scrapeunblocker) for targets that still block. Add each layer only when a site needs it, because a real browser or a paid API costs more per page than a plain HTTP request.

Which Python tool handles which layer?

Each layer has a standard open-source tool, described here in its own documentation’s terms:

Layer	Tool	What it does
Crawling	Scrapy	Open-source web scraping framework for Python; the scrapy-playwright extension adds browser rendering
Parsing	Beautiful Soup	Navigates and searches the parse tree; sits on top of parsers such as lxml and html5lib
Browser-like HTTP	curl_cffi	Impersonates browsers’ TLS/JA3 and HTTP/2 fingerprints; needs Python 3.10 or later
JavaScript pages	Playwright for Python	Automates Chromium, WebKit and Firefox, with sync and async APIs
Hard targets	ScrapeUnblocker	Rendered HTML or JSON per request; Python 3.8 or later, one dependency (httpx)

Library walkthrough: Essential Python libraries for web scraping.

How do you call ScrapeUnblocker from Python?

Install the package, set SCRAPEUNBLOCKER_KEY in your environment, and call the client:

from scrapeunblocker import Client, BlockedError

su = Client()  # reads SCRAPEUNBLOCKER_KEY
try:
    html = su.get_page_source("https://example.com")
    product = su.get_parsed("https://www.amazon.com/dp/B08N5WRWNW")
except BlockedError:
    pass  # 403: blocked on every bypass path, not billed

The client has sync and async versions (AsyncClient), retries 429, 502, 503 and 504 responses with exponential backoff, and raises typed errors. get_parsed returns JSON built from Schema.org data, __NEXT_DATA__ or AI-generated rules. Pass the HTML to Beautiful Soup or your Scrapy selectors as usual. Reference: Python SDK docs.

How do you add ScrapeUnblocker to a Scrapy project?

Install scrapeunblocker-scrapy-middleware and enable it in settings.py:

DOWNLOADER_MIDDLEWARES = {
    "scrapeunblocker_middleware.ScrapeUnblockerMiddleware": 543,
}
SCRAPEUNBLOCKER_API_KEY = "YOUR_API_KEY"

The middleware rewrites each request to /getPageSource and restores the original URL on the response, so your spiders and selectors stay unchanged. Per-request options such as proxy_country or parsed_data go in Request.meta["scrapeunblocker"]. To pay only for pages that are actually blocked, route just the failed requests through the API: Add a scraping API fallback to Scrapy.

Which other scraping APIs have official Python clients?

Several do. According to their documentation, ScraperAPI publishes scraperapi-sdk, Zenrows has SDKs for Python, Node.js and Go, ScrapingBee shows Python and Node.js packages, Bright Data has Python and JavaScript SDKs, and Scrapfly lists Python, TypeScript, Go and Rust SDKs plus Scrapy. ScrapeUnblocker’s official libraries cover Python, Node.js, Ruby and PHP. Choose by what your targets need and what each good page costs, not by the client library.

Sources

• Scrapy: scrapy.org
• Beautiful Soup: crummy.com/software/BeautifulSoup
• curl_cffi (official repository): github.com/lexiforest/curl_cffi
• Playwright for Python: playwright.dev/python/docs/intro
• ScrapeUnblocker Python SDK and Scrapy middleware: docs.scrapeunblocker.com/sdks/python, docs.scrapeunblocker.com/sdks/scrapy
• ScraperAPI SDKs: docs.scraperapi.com/resources/sdks
• Zenrows SDKs: docs.zenrows.com/fetch/sdk/overview
• ScrapingBee documentation: scrapingbee.com/documentation
• Bright Data documentation: docs.brightdata.com
• Scrapfly SDKs: scrapfly.io/docs/sdk

Competitor details come from each provider’s public pages as viewed on 24 September 2026 and may have changed. Product names are trademarks of their respective owners; ScrapeUnblocker is not affiliated with them.