How to Scrape Google Search Results With Ads Using a SERP API

Ask a developer to “scrape Google” and they usually mean the ten organic blue links. But a modern search results page is a lot more than that. Above and below the organic listings sit paid ads, and for commercial queries those ads are often the most valuable data on the page. They tell you who is bidding on a keyword, what copy they are running, and which landing pages they push. If your scraper drops them, you are missing half the picture.

Capturing ads reliably is harder than capturing organic results, and most SERP tools quietly skip them - largely because Google only shows ads to visitors it believes are real people. This post explains why paid results are so slippery, what a complete SERP response should contain, and how to pull both organic and ad results as clean JSON - the way a real person in a specific country would actually see the page.

Why Ads Are Harder to Scrape Than Organic Results

Organic results are relatively stable. Rankings shift, but the same query from two similar machines usually returns a similar list. Ads do not behave that way, and there are a few reasons why.

Google simply does not serve ads to a visitor it does not trust. This is the single biggest reason, and it is the one almost nobody mentions. Every ad impression costs an advertiser money, so Google protects its inventory far more aggressively than it protects the organic listings. If the session does not look like a genuine human - browser fingerprint, IP reputation, cookies, timing, interaction signals - the ad auction is quietly skipped. You do not get a block page, a captcha, or an error. You get a perfectly normal-looking results page with the organic listings all present, and no sponsored blocks at all.

That distinction matters: getting past the block is not the same as being trusted. Plenty of automated setups load Google successfully, get a clean HTTP 200, parse the organic results without a hitch, and still never see a single ad - for any keyword, in any country. Teams then conclude “there were no ads for this query” when a real user in the same market is looking at four of them. Because there is no error to catch, this failure is invisible unless you compare against a real browser by hand.

Earning that trust is not a parsing problem, it is an infrastructure problem: a real browser stack with a consistent fingerprint, a residential exit that matches the target country, believable session and timing behaviour, and none of the automation tells that Google has spent years learning to spot. That is exactly the layer ScrapeUnblocker handles - requests arrive as a genuine user from the country you pick, so the ad auction runs normally and the sponsored blocks come back in full, the same ones a person sitting in that market would see.

Ads are heavily personalized by location. Google’s ad auction runs per query, per region, sometimes per city. A search for “car insurance” from a US IP surfaces a completely different set of advertisers than the same search from Germany or the UK. If your scraper exits from a random datacenter IP in the wrong country, you either see the wrong ads or none at all.

Ads render late. Many ad slots are filled by client-side JavaScript a beat after the main page loads. A naive HTTP fetch that grabs the initial HTML often captures the page before the ad auction has painted, so the results look ad-free even when they are not.

Ads are designed to look like organic results. Google has spent years making the “Sponsored” label subtle. To a parser, an ad block and an organic block can look almost identical in the DOM, which means brittle, hand-written selectors miss them or misclassify them constantly.

Ad markup changes often. Google reshuffles its SERP layout frequently. Selectors that worked last month silently return empty arrays this month, and you do not find out until your dashboard goes quiet.

Put together, these factors are why so many SERP scrapers return organic-only data. Getting ads too means looking like a real user first, and then solving geo-targeting, JavaScript rendering, and robust classification on top of that.

What a Complete SERP Response Should Contain

A SERP API that is worth using should hand you the whole page, already split into the parts you care about. At minimum, that means three buckets:

• topAds - the sponsored results shown above the organic listings.
• organic - the standard, unpaid results.
• bottomAds - the sponsored results shown below the organic listings.

Each result, whether paid or organic, should carry the same clean shape: a title, the real destination URL, the displayed (pretty) URL, a description snippet, and a position. When ads and organic results share one schema, your downstream code stays simple - you loop over them the same way and just track which bucket they came from.

Crucially, the results should reflect a real user in a chosen country. You pick the country, and the API returns the page as someone browsing from there would see it - the same advertisers, the same order, the same local nuances. That is the difference between “some Google results” and “the results your target audience actually sees.”

Making the Request

The endpoint is a single POST call. The only required field is your search keyword; everything else is optional and gives you control over depth and location.

• keyword (required) - the search query, for example best running shoes.
• proxy_country - the ISO country code for geographic targeting, such as us or de.
• pages_to_check - how many SERP pages to collect, from 1 to 10 (defaults to 1).
• wait_after_load - seconds to wait after the page loads so late-rendering ads have time to appear.
• captcha_pause - seconds to pause if a challenge is encountered.

Here is the simplest possible call with curl:

curl -X POST "https://api.scrapeunblocker.com/serpApi" \
  -H "x-scrapeunblocker-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "keyword": "best running shoes",
    "proxy_country": "us",
    "pages_to_check": 1,
    "wait_after_load": 3
  }'

The two parameters that matter most for ad capture are proxy_country and wait_after_load. Set the country to wherever your audience is, and give the page a couple of seconds so the ad slots finish rendering before the data is read. Skipping either one is the most common reason people get incomplete ad data.

The same call in Python is just as short:

import requests

resp = requests.post(
    "https://api.scrapeunblocker.com/serpApi",
    headers={"x-scrapeunblocker-key": "YOUR_API_KEY"},
    json={
        "keyword": "best running shoes",
        "proxy_country": "us",
        "pages_to_check": 1,
        "wait_after_load": 3,
    },
)

data = resp.json()
print(f"{len(data['topAds'])} top ads, {len(data['organic'])} organic results")

The Response

The response is one JSON object. The top level tells you what was searched and from where, and the topAds, bottomAds, and organic arrays hold the results. Here is an example with placeholder data:

{
  "keyword": "best running shoes",
  "proxy_country": "us",
  "totalResults": 128000000,
  "topAds": [
    {
      "title": "SwiftSole Pro 2026 - Free 60-Day Returns",
      "url": "https://www.swiftsole.example/pro-running",
      "displayedUrl": "swiftsole.example/pro",
      "description": "Lightweight carbon-plate trainers built for race day. Free shipping both ways.",
      "position": 1
    },
    {
      "title": "PaceRunner Outlet - Up to 40% Off",
      "url": "https://shop.pacerunner.example/sale",
      "displayedUrl": "shop.pacerunner.example/sale",
      "description": "Last-season road and trail shoes at clearance prices. Sizes selling fast.",
      "position": 2
    }
  ],
  "bottomAds": [
    {
      "title": "TrailheadGear - Trail Running Specialists",
      "url": "https://www.trailheadgear.example/shoes",
      "displayedUrl": "trailheadgear.example",
      "description": "Grippy, waterproof trail shoes tested on real mountain routes.",
      "position": 1
    }
  ],
  "organic": [
    {
      "title": "The 12 Best Running Shoes of 2026, Tested",
      "url": "https://www.runreview.example/best-running-shoes",
      "displayedUrl": "runreview.example > gear > shoes",
      "description": "We logged 400 miles in this year's top trainers. Here is what held up.",
      "position": 1
    },
    {
      "title": "How to Choose Running Shoes - A Beginner's Guide",
      "url": "https://www.stridelab.example/guides/choose-shoes",
      "displayedUrl": "stridelab.example/guides",
      "description": "Pronation, drop, cushioning: the three things that actually matter.",
      "position": 2
    }
  ]
}

Because every entry - ad or organic - uses the same five fields, iterating is trivial. You can flatten all three arrays, tag each row with its source, and drop them straight into a table:

rows = []
for bucket in ("topAds", "bottomAds", "organic"):
    for item in data[bucket]:
        rows.append({"source": bucket, **item})

What You Can Build With Full SERP Data

Once you have organic and ad data together, several use cases open up that organic-only scrapers cannot serve:

• Competitive ad monitoring. Track which companies bid on your core keywords, watch their ad copy change over time, and spot new entrants the day they start advertising.
• Rank tracking that reflects reality. The organic position that matters is the one a user sees after ads push listings down the page. Capturing ads lets you measure true visibility, not a theoretical rank.
• Brand protection. Detect competitors or affiliates bidding on your brand name in specific countries, which is often the first sign of trademark misuse.
• Market and pricing research. Ad copy is a live feed of what promotions, discounts, and messaging your market is running right now.
• Local intent analysis. Because you choose the country, you can compare how the same query looks across markets and tailor campaigns per region.

The common thread is geography. None of this works if you cannot reliably see the page as a local user does, which is exactly why country targeting and complete ad capture belong together.

Frequently Asked Questions

Why does my scraper return no ads even for commercial keywords? Most often because Google did not consider the session a real user and skipped the ad auction entirely - the page still loads and the organic results still parse, so nothing looks broken. The other two usual causes are exiting from the wrong country, so no local ads are served, and reading the page before the ad slots finished rendering. Set proxy_country to your target market and add a short wait_after_load, but the trust layer has to be solved first, and that is not something selectors or timeouts can fix.

My scraper is not getting blocked at all - why do I still see no ads? Because avoiding the block and being treated as a real visitor are two different bars, and ads sit behind the higher one. Google will happily hand an automated-looking client a clean HTTP 200 with the full organic list and simply leave the sponsored blocks out. That is why an ad-free response is not proof that the keyword has no advertisers - check the same query in a normal browser from the same country and you will usually find them. ScrapeUnblocker requests pass as genuine users, so the ads are there.

Can I collect ads from more than the first page? Yes. Use pages_to_check to pull up to ten SERP pages in one call. Bottom-of-page ads and deeper organic listings appear as you go further down the results.

How do I tell ads apart from organic results? The response already separates them for you. Sponsored results arrive in topAds and bottomAds, and unpaid results in organic, so you never have to guess based on markup or a “Sponsored” label.

Will the results match what a real user sees? That is the goal. By selecting a proxy_country, you get the page as a genuine visitor from that country would see it - the same advertisers, ordering, and localized results, rather than a generic or datacenter-skewed view.

Bringing It Together

Complete search data means the whole page: organic listings and the ads that surround them, seen from the country that matters to you. That combination is what turns a SERP scraper from a novelty into something you can build a competitive-intelligence or rank-tracking product on.

If you would rather not maintain geo-targeting, JavaScript rendering, and ad-classification logic yourself, the ScrapeUnblocker SERP API returns organic results and both ad blocks as structured JSON in a single call, with the exit country of your choosing. You can read the full parameter and response reference in the developer documentation and start testing queries from the dashboard. Whatever tooling you choose, the principle is the same: if you want the truth about a search results page, capture all of it - ads included.