How to Scrape Images From a Website: Lazy Loading, srcset and the Logo Trap

You point your scraper at a property listing to collect the apartment photos. The script runs without errors and returns three image URLs: the agency logo, a headshot of the agent, and a 1x1 grey GIF. The twelve photos of the apartment you saw in your browser are nowhere in the output.

Nothing blocked you. The page loaded fine. The problem is that <img src="..."> is the least reliable place to look for an image on a modern website. The real URLs sit in other attributes, in structured data, in JSON, or they only appear after JavaScript runs. And the page is full of images you do not want.

This guide walks through where image URLs actually hide, how to pick the right ones, and how to download them without getting HTML or a 403 back.

Why img src Lies to You

A naive image scraper does soup.find_all("img") and reads src. That breaks for five common reasons:

• Lazy loading. Many sites put a placeholder in src (a base64 data:image/gif, a spinner, or blank.gif) and keep the real URL in data-src, data-lazy-src, data-original or data-srcset. A script swaps them when the image scrolls into view. Your HTTP client never scrolls.
• Responsive images. srcset lists several sizes of the same picture, and <picture> adds <source> elements per format. The src fallback is often the smallest version.
• JavaScript galleries. A carousel may render only the first slide in HTML. The rest of the photos arrive as JSON and get injected later.
• CSS backgrounds. Some galleries use <div style="background-image: url(...)"> instead of <img> at all.
• Noise. Logos, avatars, agent photos, icons, badges, ad creatives, tracking pixels and “similar listings” thumbnails are all <img> tags too. On a listing page they can outnumber the photos you want.

The fix is a short checklist, in this order.

Step 1: Check the Structured Data First

Before you parse a single <img>, look for images the site already describes in machine-readable form. They are usually the main content images, in full size, and cleaner than anything in the layout.

• JSON-LD. Product, listing and article pages often include a <script type="application/ld+json"> block with an image field, sometimes a full array of gallery URLs.
• Open Graph. <meta property="og:image"> is the image used for link previews. It is usually the main photo, but on some sites it is the brand logo, so verify it.
• Framework state. Next.js and similar frameworks embed page data in __NEXT_DATA__ or a window.__INITIAL_STATE__ object. Gallery URLs are often there in full.

import json
import requests
from bs4 import BeautifulSoup

html = requests.get(url, headers={"User-Agent": UA}, timeout=20).text
soup = BeautifulSoup(html, "html.parser")

def jsonld_images(soup):
    found = []
    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue
        for item in data if isinstance(data, list) else [data]:
            img = item.get("image") if isinstance(item, dict) else None
            if isinstance(img, str):
                found.append(img)
            elif isinstance(img, list):
                found += [i if isinstance(i, str) else i.get("url") for i in img]
            elif isinstance(img, dict):
                found.append(img.get("url"))
    return [u for u in found if u]

If the gallery is loaded from a separate request, open DevTools, filter the Network tab by Fetch/XHR and look for a response full of image URLs. We cover that technique in detail in how to find the hidden JSON API behind a JavaScript website.

Step 2: Scope to the Gallery, Not the Whole Page

This is the fix for the logo trap. Do not collect every image on the page and filter afterwards. Find the element that wraps the photo gallery and only search inside it.

Open the page in your browser, right-click a gallery photo, choose Inspect, and walk up the DOM until you hit the container: something like a div with a class containing gallery, carousel, slider or media, or an element with itemprop="image". Use that as your root.

gallery = soup.select_one("[class*=gallery], [class*=carousel], [data-testid*=gallery]")
root = gallery or soup.select_one("main") or soup
imgs = root.find_all("img")

The broad selectors above are a starting point, not a rule. Every site names things differently, so pin the selector to the real container once you have found it. The agent card, the header logo and the “similar listings” strip live outside the gallery, so they drop out automatically.

Step 3: Read Every Attribute That Can Hold a URL

Inside the gallery, check the lazy-loading attributes before src, and pick the largest candidate from any srcset. Resolve relative and protocol-relative URLs (//cdn.example.com/...) against the page URL.

import re
from urllib.parse import urljoin

LAZY_ATTRS = ["data-src", "data-lazy-src", "data-original", "data-url", "src"]
SRCSET_ATTRS = ["data-srcset", "srcset"]

def largest_from_srcset(srcset):
    best, best_size = None, -1
    # split on ", " or right after a descriptor, so CDN URLs with commas survive
    for part in re.split(r",\s+|(?<=\d[wx]),", srcset.strip()):
        bits = part.strip().split()
        if not bits:
            continue
        size = 1.0
        if len(bits) > 1 and bits[1][-1] in "wx":
            try:
                size = float(bits[1][:-1])
            except ValueError:
                pass
        if size > best_size:
            best, best_size = bits[0], size
    return best

def image_url(tag, page_url):
    for attr in SRCSET_ATTRS:
        best = largest_from_srcset(tag.get(attr, ""))
        if best:
            return urljoin(page_url, best)
    for attr in LAZY_ATTRS:
        value = tag.get(attr, "")
        if value and not value.startswith("data:"):
            return urljoin(page_url, value)
    return None

urls = [u for u in (image_url(img, url) for img in imgs) if u]

# <picture><source srcset="..."> and CSS background images
for source in root.select("picture source[srcset]"):
    urls.append(urljoin(url, largest_from_srcset(source["srcset"])))
for el in root.select("[style*=background-image]"):
    m = re.search(r"url\(['\"]?(.*?)['\"]?\)", el["style"])
    if m:
        urls.append(urljoin(url, m.group(1)))

Note that srcset uses two kinds of descriptors: width (800w) and pixel density (2x). Both work with the “pick the biggest number” rule as long as a single attribute does not mix them, which the HTML spec does not allow anyway.

Step 4: Filter Out the Noise

Even inside the gallery you will catch some junk. Filter it in two passes.

Before download, using the URL and attributes:

• Drop data: URIs and .svg files (icons and logos are almost always SVG).
• Drop URLs or class names containing words like logo, avatar, icon, sprite, placeholder, badge or pixel.
• Drop images with tiny width or height attributes, for example under 100 pixels.
• Deduplicate. The same photo often appears twice: once as a thumbnail strip and once as the main slide.

After download, using the actual file:

• Open it with Pillow and discard anything below a minimum size, such as 300 pixels on the short side. This catches placeholders that slipped through.
• For near-duplicates (the same photo at two sizes or compressions), compare perceptual hashes with the imagehash library rather than byte hashes.

from io import BytesIO
from PIL import Image
import imagehash

def keep(content, seen_hashes, min_side=300):
    img = Image.open(BytesIO(content))
    if min(img.size) < min_side:
        return False
    h = imagehash.phash(img)
    if any(h - other <= 4 for other in seen_hashes):
        return False
    seen_hashes.append(h)
    return True

Step 5: Upgrade Thumbnails to Full Size

Sometimes the HTML only holds thumbnails, and the full-size image is loaded when a user opens the lightbox. Many image CDNs encode the size in the URL, either as a query parameter (?w=320, ?width=320&quality=60) or as a path segment (/320x240/, _thumb).

Do not guess the pattern. Open the lightbox in your browser, switch the Network tab filter to Img, and compare the URL of the big image with the thumbnail you scraped. Usually one segment changes. Once you know the rule for that site, apply it to every thumbnail URL. If the CDN rejects sizes it does not know, stick to the exact values the site itself requests.

Step 6: When the Gallery Only Exists After JavaScript

If the photos are not in the HTML, the JSON-LD or an API response you can call, render the page. With Playwright, the most robust trick is to listen to the network instead of the DOM: every image the browser loads passes through as a response, whatever attribute or script put it there.

from playwright.sync_api import sync_playwright

image_urls = []

def on_response(response):
    if response.request.resource_type == "image" and response.ok:
        image_urls.append(response.url)

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.on("response", on_response)
    page.goto(url, wait_until="domcontentloaded")
    for _ in range(8):  # scroll to trigger lazy loading
        page.mouse.wheel(0, 1500)
        page.wait_for_timeout(500)
    html = page.content()
    browser.close()

Network capture collects logos and icons too, so run the same filters from Step 4. Combine it with Step 2: the rendered html lets you scope to the gallery and read the final attributes, while the network list catches images that never appeared in the DOM at all. If a carousel loads slides only when you click “next”, click through it before you collect. Playwright’s own guide to input actions covers clicks, typing and other interactions in more depth.

Step 7: Download Like a Browser

The last trap is the download itself. Image hosts are often stricter than the page.

• Send a Referer header with the page URL. Hotlink protection checks it and returns a 403 or a substitute image without it.
• Send a normal User-Agent and an image Accept header, for example image/avif,image/webp,image/*,*/*;q=0.8.
• Check Content-Type. If it starts with text/html, you got an error page or a challenge, not an image. Take the file extension from the content type, because a URL ending in .jpg may serve WebP or AVIF.
• Cache by URL and never download the same image twice. Image requests are heavy, and hammering a CDN is the fastest way to get rate limited.
• Throttle. A few parallel downloads with a small delay is plenty for most jobs.

import hashlib

headers = {
    "User-Agent": UA,
    "Referer": url,
    "Accept": "image/avif,image/webp,image/*,*/*;q=0.8",
}
r = requests.get(img_url, headers=headers, timeout=30)
ctype = r.headers.get("Content-Type", "")
if r.ok and ctype.startswith("image/"):
    ext = ctype.split("/")[1].split(";")[0].replace("jpeg", "jpg")
    name = hashlib.sha1(img_url.encode()).hexdigest()[:16]  # stable name = free cache key
    with open(f"images/{name}.{ext}", "wb") as f:
        f.write(r.content)

Photos are usually protected by copyright. Collecting URLs or metadata for analysis is a different thing from republishing someone else’s pictures, so check the site’s terms and your use case. Our guide to web scraping legality covers the basics. It is not legal advice.

FAQ

Why does BeautifulSoup give me a base64 GIF instead of the image URL? The site uses lazy loading. The src holds a tiny placeholder and the real URL is in an attribute like data-src or data-srcset. Read those first, as shown in Step 3.

How do I get the largest image from srcset? Split the attribute on commas, read the descriptor after each URL (800w or 2x), and take the URL with the highest number. If the largest size is still small, look for the lightbox URL pattern described in Step 5.

Why do my downloaded “images” open as HTML? The image host returned an error page, a login redirect or an anti-bot challenge instead of the file. Add a Referer and browser-like headers, check Content-Type before saving, and slow down.

Do I need a headless browser to scrape images? Often not. Structured data, lazy-loading attributes and the site’s own JSON cover most pages with plain HTTP requests. Use a browser when the gallery is built entirely by JavaScript or only loads on interaction.

Wrapping Up

Scraping images is mostly a parsing problem, not an access problem. Check structured data first, scope your selector to the gallery, read the lazy-loading and srcset attributes, filter by size and perceptual hash, and download with the headers a browser would send. That combination turns “a logo and a grey GIF” into the full set of photos.

When access does become the problem, ScrapeUnblocker can help on both ends. It returns the rendered HTML of a page, and its browser steps can scroll or click through a gallery before the HTML comes back. For image hosts that answer a plain request with a challenge page or a tracking pixel, the image fetching endpoint loads the file through a real browser and returns the image bytes.