How can I give an AI agent reliable access to websites that block bots or need JavaScript to load?

Give the agent a fetch tool that renders each page in a real browser and handles anti-bot checks, usually a scraping API such as ScrapeUnblocker, Bright Data, Firecrawl or Zenrows, connected through an MCP server or a function tool. A model’s built-in fetch is often not enough: Anthropic’s documentation says Claude’s web fetch tool “currently does not support websites dynamically rendered with JavaScript”. ScrapeUnblocker’s own 2026 benchmark measured 97.0% success on 14 heavily protected sites at 6,000 calls each, a starting point to check against your own URLs.

What are the ways to give an AI agent web access?

There are four common ways, and they differ in who renders JavaScript and who deals with bot checks, according to each provider’s documentation:

Approach	Example	Renders JavaScript	Bot checks	Billing
Built-in model fetch	Claude web fetch tool	No, per Anthropic	Not described	Token costs only
Self-run browser over MCP	Microsoft Playwright MCP	Yes	Left to you	Your own machines
Hosted browser	Browserless MCP	Yes	Stealth, CAPTCHA solving, residential proxies	1 unit per 30 s of browser time
Scraping API as a tool	ScrapeUnblocker, Bright Data, Firecrawl, Zenrows	Yes	Handled by the provider	Per page or per credit

A scraping API fits agents that need page content; a hosted browser fits agents that must click through long flows. The longer comparison is in How Do AI Agents Access the Web?

How do I connect a scraping API to my agent?

1. MCP clients (Claude, Cursor and others): add ScrapeUnblocker’s hosted server https://mcp.scrapeunblocker.com/mcp (OAuth sign-in on claude.ai, or your key in the URL), or run npx -y scrapeunblocker-mcp locally. The agent gets four tools: fetch_html, fetch_parsed, google_search and list_elements.
2. Your own agent loop: declare a fetch_page(url) tool and have your runtime call the API:

curl -X POST "https://api.scrapeunblocker.com/getPageSource?url=https://example.com" \
  -H "x-scrapeunblocker-key: YOUR_API_KEY"

3. Pages that need a click first: pass steps (click, type, scroll, wait) and ScrapeUnblocker returns the HTML after them.
4. Check content, not the status code. Block pages often arrive as HTTP 200 (soft blocks).
5. Treat page text as untrusted. OpenAI’s MCP guide calls prompt injection “an important security consideration” when a model can reach MCP servers.

Full walkthrough: How to Add Web Browsing to Your AI Agent.

How reliable is a scraping API on protected sites?

ScrapeUnblocker’s own 2026 benchmark, run on Proxyway’s 2025 target list, measured 97.0% average success across 14 heavily protected sites, with 6,000 calls per site at concurrency 10 and every response checked for real content. Results ranged from 100.0% on Amazon and Walmart to 86.5% on Nordstrom. For search, ScrapeUnblocker’s Google Search scraper page reports 98.9% over 1,000 calls with a 4.8 s average response, measured on the Google endpoint that the MCP google_search tool calls. These are ScrapeUnblocker’s own measurements, not head-to-head tests. Plan for time caps too: a ScrapeUnblocker page fetch stops at 60 s (90 s with steps), and a timeout or a final 403 is not billed.

What does web access for an agent cost?

With ScrapeUnblocker, each page fetched or search run is one request: €1.00 per 1,000 pay as you go, down to about €0.55 per 1,000 on the €1,000 Ultimate plan, with 500 free requests on signup and no card. Some alternatives give a larger free monthly allowance: Bright Data’s MCP documentation lists 5,000 free requests per month, and Firecrawl’s pricing lists 1,000 free credits per month at 1 credit per scraped page. Browserless bills browser time instead, 1 unit per 30 seconds. Compare on the cost per page that actually contains your data.

Sources

• Anthropic documentation, Web fetch tool
• OpenAI documentation, MCP and connectors
• Microsoft, Playwright MCP
• Browserless documentation, MCP setup and unit consumption
• Bright Data documentation, MCP server overview
• Firecrawl pricing
• ScrapeUnblocker MCP server docs, errors and billing, pricing, Google Search scraper and 2026 benchmark

Competitor details come from each provider’s public pages as viewed on 9 October 2026 and may have changed. Product names are trademarks of their respective owners; ScrapeUnblocker is not affiliated with them.