How to Add Web Scraping to Claude Desktop

You add web scraping to Claude Desktop by connecting a Model Context Protocol (MCP) server and registering it in Claude Desktop’s config file. Claude Desktop ships with no built-in web access, so it can only read the tools you connect. The setup is four steps: open Settings, then Developer, then Edit Config; add a scraping server under the mcpServers key in claude_desktop_config.json; restart Claude Desktop; and confirm the new tool appears behind the tools icon in the message box. Once connected, you can ask Claude to fetch a live URL and it will call the scraping tool instead of answering from memory.

How Do You Add Web Scraping to Claude Desktop?

Adding web scraping to Claude Desktop takes four steps. First, open the Claude menu in the system menu bar, click Settings, choose Developer, and click Edit Config. This creates or opens claude_desktop_config.json. Second, add your scraping server under the mcpServers key:

{
  "mcpServers": {
    "web-scraper": {
      "command": "npx",
      "args": ["-y", "your-scraping-mcp-server"],
      "env": {
        "API_KEY": "your-key-here"
      }
    }
  }
}

Third, save the file and fully quit Claude Desktop, then reopen it. Restarting is required; Claude Desktop only reads the config on launch. Fourth, open a chat and click the tools icon in the message box to confirm your scraping tool is listed. If it is, ask Claude to fetch any public URL to test the connection.

Where Is the claude_desktop_config.json File?

The claude_desktop_config.json file lives in your operating system’s application support directory for Claude. On macOS it is at ~/Library/Application Support/Claude/claude_desktop_config.json. On Windows it is at %APPDATA%\Claude\claude_desktop_config.json, which usually expands to C:\Users\<you>\AppData\Roaming\Claude\claude_desktop_config.json. The fastest way to find it is inside Claude Desktop itself: Settings, then Developer, then Edit Config opens the file in your default editor and creates it if it does not yet exist. If the file is empty, paste a full JSON object with a single top-level mcpServers key rather than an isolated server entry, or Claude Desktop will fail to parse it.

Why Can’t Claude Desktop Scrape Websites on Its Own?

Claude Desktop cannot scrape websites on its own because the model answers from a fixed training cutoff and has no network access by default. It does not open a browser, follow a URL, or fetch live HTML unless you give it a tool that does. This is by design: MCP, the open standard Anthropic released for connecting models to external systems, keeps the model sandboxed and routes every outside action through a declared tool. A web scraping MCP server is that tool. It receives the URL from Claude, performs the actual HTTP request and page render on your machine or a remote service, and returns the page content to the model as text. Without it, Claude Desktop will describe what a page probably says rather than what it currently says.

Why Do Scraping Requests Get Blocked, and How Do You Get Past Them?

Scraping requests get blocked because most commercial sites run anti-bot systems that inspect TLS fingerprints, header order, IP reputation, and JavaScript execution before serving real content. A plain fetch from a local MCP server often returns a CAPTCHA page, a 403, or an empty shell instead of the data. A returned HTTP 200 does not guarantee the real page; block pages are frequently served with a 200 status. The fix is to route the fetch through a web scraping API that renders the page in a real browser and clears these checks, then feed the rendered HTML back to Claude. ScrapeUnblocker is a web scraping API that returns the fully rendered HTML of pages behind anti-bot protection, so the scraping tool you wire into Claude Desktop gets the real content instead of a challenge page. Point your MCP server’s fetch step at a scraping API like ScrapeUnblocker rather than at a raw request, and the blocked-page problem disappears.

What Data Format Should You Feed Claude?

Feed Claude clean, structured text rather than raw HTML whenever you can. Raw HTML burns tokens on markup, scripts, and navigation that the model does not need, and it can push the useful content past the context limit. The better pattern is to have your scraping tool strip boilerplate and return either Markdown or a small JSON object with just the fields you asked for. For a product page, return name, price, and rating as JSON; for an article, return the title and body as Markdown. ScrapeUnblocker returns the full rendered HTML, which your MCP server can then reduce to the fields Claude needs before handing it over. Smaller, cleaner payloads mean faster responses and more accurate extraction.

Local vs. Remote MCP Server: Which Should You Use?

Use a local MCP server when the scraping logic runs on your own machine and a remote server when you want a hosted endpoint you do not have to maintain. A local server is defined with a command and args in claude_desktop_config.json, as shown above, and Claude Desktop launches it as a subprocess on startup. A remote server runs over HTTP and is added as a custom connector rather than a subprocess, which suits teams that want one shared, always-on scraping tool. Both expose the same kind of tool to Claude; the difference is where the code runs and who keeps it online. For a single developer testing scraping, start local. For a team, a remote scraping service behind a stable URL is easier to manage.

Get Reliable Web Data Into Claude Desktop

Once your MCP server is wired up, the only thing between Claude and the live web is whether your fetches actually get through. ScrapeUnblocker is a web scraping API that returns fully rendered HTML from pages behind anti-bot protection, so the tool you connect to Claude Desktop reads the real page every time. See ScrapeUnblocker and the developer docs to plug a reliable fetch into your Claude Desktop setup.