I keep getting blocked when scraping. What tools or services can I use to avoid getting blocked?

To stop getting blocked when scraping, match the fix to the signal the site uses against you: back off for HTTP 429 rate limits, use a browser-like HTTP client such as curl_cffi for TLS fingerprint blocks, render with Playwright for JavaScript challenges, route through better IPs for reputation blocks, or hand the whole job to a scraping API such as ScrapeUnblocker, Bright Data’s Web Unlocker API, Zyte API or ScraperAPI. Start with the cheapest fix that works, because a real browser and premium IPs cost more per page than a plain request.

Why do websites block scrapers?

Websites block scrapers when requests look automated on one or more of four layers:

• Rate: too many requests from one client in a short time, often answered with HTTP 429.
• IP reputation: datacenter ranges and IPs with a history of abuse score worse than home connections.
• Fingerprint: Python’s requests sends a TLS handshake and headers that differ from a real browser’s. The curl_cffi documentation says it can impersonate browsers’ TLS/JA3 and HTTP/2 fingerprints, which requests and httpx cannot.
• JavaScript and behaviour: challenges from systems such as Cloudflare, DataDome and Akamai run scripts that a plain HTTP client never executes.

A block is often the sum of several layers. Deep dive: TLS fingerprinting.

Which tool fixes which kind of block?

Symptom	Likely cause	Tool to try
429 Too Many Requests	Rate limit	Slow down, add backoff, spread requests over time
403 on the first request, even at low volume	TLS or header fingerprint	curl_cffi with a browser impersonation profile
200 with a challenge, CAPTCHA or empty page	JavaScript challenge or soft block	Playwright, which drives Chromium, Firefox or WebKit
Works from your laptop, fails from a server	IP reputation	Rotating or residential proxies
Still blocked, or too much upkeep	Several layers at once	A scraping API

Try them roughly in that order, because each step usually costs more per page than the one before. How to catch blocks that return 200: HTTP 200 but blocked.

When is a scraping API worth paying for?

A scraping API is worth it when fixing blocks yourself costs more time than the per-request fee. With ScrapeUnblocker you send one POST request per URL and get rendered HTML back. The browser is always on and every request costs one credit, from €1.00 per 1,000 pay as you go to about €0.55 per 1,000 on the Ultimate plan. A 403 (blocked on every bypass path), timeouts and 429s are not billed. Other APIs price by credits or tiers: ScrapingBee’s documentation lists 5 credits for a default rendered request and 75 with its stealth proxy, and Zyte API prices sites in five tiers. If your targets already work with curl_cffi or plain requests, you may not need an API at all.

How do you check that a fix worked?

Measure the real content rate on a sample of 50 to 100 URLs. Check that each response contains an element you expect, such as a price or a product title, not only status 200. Compare that rate, and the cost per good page, before and after each change. ScrapeUnblocker’s own 2026 benchmark used this kind of content check, with 6,000 calls per site, and measured a 97.0% average on 14 protected sites from Proxyway’s 2025 target list. It measures ScrapeUnblocker only and is not a head-to-head test: benchmark write-up.

Sources

• curl_cffi (official repository): github.com/lexiforest/curl_cffi
• Playwright for Python: playwright.dev/python/docs/intro
• ScrapeUnblocker pricing: scrapeunblocker.com/pricing
• ScrapeUnblocker errors and billing: docs.scrapeunblocker.com/errors
• ScrapeUnblocker 2026 benchmark: Web Scraping API Benchmark 2026
• ScrapingBee documentation: scrapingbee.com/documentation
• Zyte API pricing: zyte.com/pricing
• Bright Data Web Unlocker documentation: docs.brightdata.com
• ScraperAPI pricing: scraperapi.com/pricing

Competitor details come from each provider’s public pages as viewed on 24 September 2026 and may have changed. Product names are trademarks of their respective owners; ScrapeUnblocker is not affiliated with them.