The 10,000 Result Cap: How to Scrape Past Search Pagination Limits

You build a scraper against a search results page. It works. You paginate: page 1, page 2, page 10, page 40. New items keep coming. Then somewhere around page 65, the results dry up. The page still loads. The layout is intact. There is no CAPTCHA and no block message. There are simply no more listings, even though the site’s own header says “48,000+ results.”

You did not get banned. You hit a result cap. Almost every large search interface has one, and if you do not plan for it, your dataset silently ends at a fraction of what is actually there. This post explains what the cap is, why it exists, how to tell it apart from a block, and the one reliable technique for collecting the full set anyway: partitioning the query space.

What the Result Cap Looks Like

The symptom is specific. Shallow pages return fresh data. Deep pages return an empty grid, or the same last page repeated, or a redirect back to page 1. Crucially, the response is a valid page. Status code 200. Correct title. Correct navigation. Just no items.

Two numbers usually define the cap:

• A maximum offset, often 10,000. Many sites paginate with an offset or page number that maps to page * page_size. Once offset crosses ~10,000, the backend refuses to go deeper.
• A maximum results per query, which is the same limit seen from the other side. If page size is 240, you get roughly 10,000 / 240 = 41 usable pages. If page size is 50, you get 200 pages. Either way you top out near 10,000 items.

You will see this on large e-commerce marketplaces, job boards, real-estate portals, and general web search engines. It is not a bug. It is a deliberate ceiling.

Why Sites Cap Deep Pagination

There is no conspiracy here. The cap is a performance decision, and understanding it tells you how to beat it.

Most search backends use offset pagination. To serve page 400, the database has to walk past the first 399 pages of results and then return the next slice. That “walk past” work grows linearly with depth. Page 1 is cheap. Page 4,000 is expensive. To return deep pages the engine must sort and scan an enormous candidate set, and it must do that for every crawler that walks the tail. So the operators draw a line. Beyond offset 10,000, the request is simply rejected or returns empty.

Search engines like Elasticsearch make this explicit. Its index.max_result_window defaults to exactly 10,000, and the docs tell you to use search_after or the scroll API instead of deep from/size paging. When a site is built on that stack, the 10,000 ceiling is inherited straight from the engine.

The important takeaway: the cap applies to a single query, not to the whole catalog. The catalog has millions of items. Any one query can only expose the first 10,000 of them. So the fix is not to paginate harder. The fix is to ask more, smaller questions.

First, Confirm It Is a Cap and Not a Block

Before you redesign your crawler, rule out a soft block. Empty deep pages and stealth blocks can look identical from the outside, so check three things:

1. Is the page structurally valid? A real capped page has the site’s normal header, footer, and “no results” messaging. A block often returns a stripped page, a challenge, or an interstitial. If you are unsure how to tell these apart, our write-up on detecting soft blocks covers the signals.
2. Does the boundary reproduce? Request the same deep page three times. A cap is deterministic: page 70 is always empty. A block is often intermittent, clearing after a pause or a new session.
3. Does a shallower page still work in the same session? If page 5 returns data but page 70 does not, using the same cookies and headers, you are looking at a cap, not an IP ban.

Once you confirm it is a cap, stop trying to scroll past it. No amount of proxies or retries moves a hard offset limit.

The Fix: Partition the Query Space

The whole technique is one idea. Split the catalog into slices small enough that each slice contains fewer than 10,000 results, scrape each slice fully, then merge and deduplicate. Every large search UI gives you filters, and every filter is a knife you can cut with.

Slice by Category

The most natural cut. Instead of searching “laptops” (500,000 results, capped at 10,000), search each subcategory: “gaming laptops,” “business laptops,” “chromebooks,” and so on. Category filters are almost always exposed as URL parameters, which makes them trivial to iterate.

Slice by Price Band

Price is a clean numeric axis. Most search interfaces accept a minimum and maximum price. Walk the range in bands:

def price_bands(lo, hi, step):
    edge = lo
    while edge < hi:
        yield (edge, min(edge + step, hi))
        edge += step

# 0-25, 25-50, 50-75, ...
for low, high in price_bands(0, 1000, 25):
    scrape_query(price_min=low, price_max=high)

If a single band still returns more than 10,000 results, split it further. Narrow bands near the low end, where inventory clusters, and wider bands at the top.

Slice by Date Window

For anything ordered by recency, such as job listings, news, or newly listed items, window by time. Query “posted in the last 24 hours,” step back a day, repeat. Date slicing has a bonus: it turns a one-time crawl into a maintainable incremental one. After the first full sweep, you only ever pull the newest window.

Slice by Location

Region, country, city, or postal code. This axis works well for marketplaces and directories where the same query in different locales returns largely different inventory. Do not assume the slices are disjoint, though. The same seller or listing can appear under several regions, which is exactly why the dedup step below matters.

Combine Axes When One Is Not Enough

If category alone still busts the cap, combine axes: category multiplied by price band, or location multiplied by date window. Each additional axis multiplies your slice count but shrinks each slice below the ceiling. The goal is simple: no single query should return more than ~10,000 results.

Deduplicate Across Slices

Overlapping slices will re-surface the same items. You need a stable identity key and a seen-set. Prefer the site’s own permanent identifier, usually embedded in the item URL, over anything cosmetic like a title or a display name that can change between requests.

seen = set()
results = []

for slice_query in build_slices():
    for item in scrape_all_pages(slice_query):
        item_id = extract_id(item["url"])   # stable, from the URL
        if item_id in seen:
            continue
        seen.add(item_id)
        results.append(item)

Keep the seen-set persistent across runs if you scrape on a schedule. That way an incremental sweep skips what you already have, and your call budget goes toward genuinely new items.

Watch the Marginal Return

Partitioning is powerful but not free. Each slice costs requests, and slices overlap, so you pay for duplicates. Track how many new items each slice contributes. When a slice returns almost nothing you have not already seen, you are near saturation and can stop. This “stop when dry” rule keeps you from grinding through thousands of near-empty queries chasing the last handful of long-tail items. For most projects, capturing the bulk of the catalog costs a fraction of what full coverage of every one-off entry would.

Also be gentle. Deep search pages are among the heaviest a site renders, and hammering them in parallel is the fastest way to turn a clean crawl into a wave of 503s. Keep concurrency modest on those endpoints and let the slices, not raw parallelism, do the scaling.

FAQ

Is the cap always 10,000? No, but it is the most common value because it is the Elasticsearch default. Some sites cap at 1,000, some at 25,000. Measure it: binary-search the depth where results go empty, then design slices to stay comfortably under that number.

Can I use an API to avoid the cap? Sometimes. Cursor-based APIs (search_after, scroll tokens, “next page” cursors) do not suffer the offset problem, so if a documented API offers cursor pagination, prefer it. But most public APIs apply the same per-query result ceiling, so you often still need to partition. Always check the terms of service before using an API for bulk collection.

Why not just increase the page size? Because the cap is on total results per query, not on page count. A bigger page size gets you to the ceiling in fewer requests, but the ceiling does not move. Larger pages are still worth using to cut request volume, just do not expect them to unlock more data.

How do I know I got everything? You do not, with certainty. Use the marginal-return signal: once fresh slices stop adding new items, and independent axes (say, category slicing and price slicing) converge on the same total, you have strong evidence of near-complete coverage.

Bringing It Together

Result caps are a fact of life when you scrape at scale. The instinct to paginate harder or throw more proxies at a deep page is wasted effort, because the limit is structural, not defensive. The reliable move is to partition: cut the catalog by category, price, date, and location until every query fits under the ceiling, then merge and deduplicate on a stable key.

The one thing partitioning does not solve is getting each individual page in the first place. Deep search pages are exactly where anti-bot systems concentrate, and a crawler that fires hundreds of heavy render requests will draw attention no matter how cleanly you slice. If you would rather not maintain the proxy rotation and browser fingerprinting that reliable page fetching requires, ScrapeUnblocker handles that layer and returns the rendered HTML, so you can focus on the slicing logic instead. Either way, the strategy is the same: stop fighting the cap, and start dividing the problem.