New in ScrapeUnblocker: Interact With a Page, Then Scrape It

ScrapeUnblocker is our web scraping API: you give it a URL, and it returns the fully rendered HTML of that page, getting past anti-bot protection for you along the way. Today we are adding a new capability. Before it hands you the HTML, ScrapeUnblocker can now interact with the page.

That means you can reach the data that only shows up after an action - type a query and run a search, click a “load more” button, choose a country or currency from a dropdown, or step through a multi-part form - and get back the HTML the page reached once those actions ran. A companion feature lists every interactive element on a page, so you know exactly what to click, type into, or select. This post covers both, and how to use them together.

How to use it

Both features are plain query parameters on the same page-fetching request you already make (the getPageSource endpoint). There is no new URL, no new plan, and no browser to run on your side.

• list_elements=true returns a JSON map of the page’s interactive elements instead of HTML.
• steps=[...] takes a JSON array of actions to run after the page loads, and returns the HTML the page reached.

You will almost always use them in that order: list_elements to find the selectors, steps to act on them.

Discover a page’s elements

Before you can act on a page, you need to know what is on it. Set list_elements=true and, instead of raw HTML, you get back a compact JSON list of the things you can act on - buttons, inputs, text areas, selects, links, and forms. Each one comes with a ready-to-use selector plus its text, name, placeholder, and role where those exist.

curl -X POST -G "https://api.scrapeunblocker.com/getPageSource" \
  --data-urlencode "url=https://www.example.com/search" \
  --data-urlencode "list_elements=true" \
  -H "X-ScrapeUnblocker-Key: YOUR_API_KEY"

The response looks like this:

{
  "url": "https://www.example.com/search",
  "count": 12,
  "elements": [
    {
      "tag": "input",
      "selector": "#searchInput",
      "type": "text",
      "name": "q",
      "placeholder": "Search products",
      "text": "Search products"
    },
    {
      "tag": "button",
      "selector": "button.search-submit",
      "text": "Search"
    },
    {
      "tag": "select",
      "selector": "select[name=\"country\"]",
      "name": "country",
      "text": "Country"
    }
  ]
}

Because the page is measured after a real browser renders and hydrates it, you see the controls a user actually sees, not just what is in the raw markup. The selectors are chosen to be stable - a real #id first, then a name, then a class, then a short positional path - so you can drop any of them straight into a step.

Act on the page with steps

Once you know the selectors, you attach steps: a JSON array of actions that run in order after the page loads. When the steps finish, you get the resulting HTML back - exactly as if you had performed the clicks and keystrokes yourself.

Here is a search flow: type a query into the box, press Enter, and wait for the results to appear before capturing the HTML.

curl -X POST -G "https://api.scrapeunblocker.com/getPageSource" \
  --data-urlencode "url=https://www.example.com/search" \
  --data-urlencode 'steps=[
    {"action":"type","selector":"#searchInput","value":"bmw"},
    {"action":"press_key","value":"Enter"},
    {"action":"wait_for","selector":".results"}
  ]' \
  -H "X-ScrapeUnblocker-Key: YOUR_API_KEY"

The response is the fully rendered HTML of the results page. The same pattern handles a “load more” button (click it, wait for new rows), a filter dropdown (select an option, wait for the list to refresh), or a two-field login-and-view flow.

A few things worth knowing:

• Typing is human-like. The type action enters text one character at a time with natural timing, instead of pasting it in one shot, so forms that watch for real keystrokes behave normally.
• It runs once. A request with steps may submit a form or change state, so it is executed a single time and not retried behind the scenes. Ask for exactly the sequence you want.
• There is a time budget. The whole sequence shares a bounded action budget of about 30 seconds, which is plenty for a search or a couple of clicks. Use wait_for on the element you actually need rather than a long fixed wait.

The actions you can run

Each step is an object with an action and, depending on the action, a selector and a value. The available actions are:

• wait_for - wait until an element matching selector is visible.
• wait_for_text - wait until a given piece of text appears anywhere on the page (value is the text).
• wait - a fixed pause, with value in milliseconds.
• click - click the element at selector.
• type - type value into the field at selector, character by character.
• select - choose an option in a <select> by its value.
• press_key - press a keyboard key such as Enter, Tab, or Escape.
• scroll - scroll the page, with value set to "bottom" or a pixel amount, which is how you trigger content that loads as you scroll.

Selectors are CSS by default. If you prefer, a step can carry "selector_type": "xPath" to target an element by XPath instead.

When a step fails, you still get the page

If a step cannot complete - a selector never matches, or an element never appears in time - the API does not just return an error and throw away the work. It responds with HTTP 422 and a JSON body that names exactly which step failed and why, and includes the HTML of the page in the state it reached.

{
  "error": "step_failed",
  "step_index": 2,
  "action": "wait_for",
  "reason": "Timeout 8000ms exceeded.",
  "selector": ".results",
  "html": "<!doctype html>..."
}

That step_index tells you the third step (counting from zero) is the one to fix, and the html lets you see what the page actually did - maybe the results container has a different class than you expected. A malformed steps payload, such as an unknown action or a missing selector, is caught even earlier and returns a 422 before any browser starts, so you never spend a request on a typo.

When is this useful

Anytime the data you want is not on the first render, but one or two interactions away:

• Search results behind a form. Type a query, submit it, and scrape the results page instead of guessing a search URL.
• Paginated or infinite-scroll listings. Click “load more” or scroll to the bottom, wait for the new rows, then capture a fuller page.
• Country, currency, or language selectors. Choose the option you need so the prices and content you scrape match the market you care about.
• Multi-step flows. Fill one field, move to the next, submit - and read the page you land on.
• AI agents that must figure out a page first. An agent can list the elements, decide what to do, then act - no hard-coded selectors needed.

Discover, then act: the full loop

The two features are designed to work as a pair, and this is where they shine for AI agents and automation. An agent that lands on an unfamiliar page cannot guess selectors reliably. So it does what a person does: it looks first.

1. Call list_elements=true to get the page’s controls and their selectors.
2. Decide what to do - which field to fill, which button to click - from that list.
3. Build a steps array against those exact selectors.
4. Call the endpoint with the steps and read the resulting HTML.

Because step one returns real, stable selectors rather than a screenshot the model has to interpret, the plan the agent builds in step three tends to work on the first try. And since every call already runs behind the same anti-bot handling as the rest of ScrapeUnblocker, the agent never has to think about blocks, IP rotation, or rendering - it just discovers the page and acts on it.

Both features are live now. See the full parameter reference and more examples in the integration docs at docs.scrapeunblocker.com, or view the plans at scrapeunblocker.com/pricing.