Search

ChatGPT Scraper

chatgpt.com

ChatGPT is served through a dedicated plugin endpoint rather than the generic page fetch. Give it a prompt - the question your customers would type, like 'best CRM software for small business' or 'is HubSpot worth it' - and, optionally, the country to ask from, and it opens a fresh signed-out ChatGPT session in a real browser, waits until ChatGPT has finished writing and returns the answer already parsed: the answer as plain text and as Markdown, every citation pill ChatGPT attaches to a claim (the site name and domain of its source, how many further sources the pill groups and the answer text it backs), the links written into the answer with their real URLs, the list of sites cited or linked, and whether ChatGPT searched the web. Signed-out ChatGPT shows the full URL of a cited page only when you hover the pill, so citations carry the site and domain rather than the page URL, and it does not show which model answered. ChatGPT's interface follows the visitor's location while the answer follows the language of the prompt. Each call starts a new conversation and takes 15-60 seconds, and answers vary between identical prompts, as they do for people, so monitoring means asking the same prompts on a schedule and comparing. No OpenAI account or cookies are involved.

What you can extract

• The finished answer as plain text and as Markdown (headings, lists, tables and images as ChatGPT wrote them)
• Citations: site name and domain of the source behind each citation pill, how many more sources it groups, and the answer text it backs
• Links written into the answer: title, URL (tracking tag removed) and domain
• Every site cited or linked, once, in first-seen order
• Whether ChatGPT searched the web for the answer
• Ask from a chosen country: local results follow it

Dedicated endpoints

How to scrape it

curl -X POST \
  -H "X-ScrapeUnblocker-Key: $SU_API_KEY" \
  "https://api.scrapeunblocker.com/ai/chatgpt?prompt=best%20CRM%20software%20for%20small%20business&country=US"

import requests

resp = requests.post(
    "https://api.scrapeunblocker.com/ai/chatgpt",
    headers={"X-ScrapeUnblocker-Key": SU_API_KEY},
    params={"prompt": "best CRM software for small business", "country": "US"},
)
print(resp.text)

import { ScrapeUnblockerClient } from 'scrapeunblocker';

const su = new ScrapeUnblockerClient({ apiKey: process.env.SU_API_KEY });

const answer = await su.chatgpt('best CRM software for small business', {
  country: 'US',
});
console.log(answer);

Requests are POST and take their parameters in the query string. Browser rendering always happens - there is no flag to enable.

Example response

{
  "url": "https://chatgpt.com/",
  "country": "US",
  "prompt": "is HubSpot worth it for a 5 person team",
  "answer": "Yes, HubSpot can absolutely be worth it for a 5-person team - but I’d start small and avoid paying for features you don’t need.

For a team that size, the biggest benefit isn't having a sophisticated  ...",
  "answerMarkdown": "Yes, HubSpot can absolutely be worth it for a 5-person team - but I’d start small and avoid paying for features you don’t need.

For a team that size, the biggest benefit isn't having a sophisticated  ...",
  "citations": [
    {
      "position": 1,
      "siteName": "HubSpot",
      "domain": "hubspot.com",
      "moreSources": 1,
      "citedText": "HubSpot's published pricing has a significant difference between basic CRM access and advanced marketing or sales tools. ..."
    },
    {
      "position": 2,
      "siteName": "Legal HubSpot",
      "domain": "legal.hubspot.com",
      "moreSources": 2,
      "citedText": "USD estimates based on published pricing and promotional rates; verify your exact checkout quote, seat requirements, bil ..."
    }
  ],
  "links": [
    {
      "position": 1,
      "title": "HubSpot's official pricing page",
      "url": "https://www.hubspot.com/pricing/suite",
      "domain": "hubspot.com"
    }
  ],
  "sources": [
    {
      "position": 1,
      "siteName": "HubSpot",
      "domain": "hubspot.com"
    },
    {
      "position": 2,
      "siteName": "Legal HubSpot",
      "domain": "legal.hubspot.com"
    }
  ],
  "webSearch": true,
  "model": null,
  "conversationId": "6ae39f2c-22f8-83ea-9559-a718945f4034"
}

Read the API docs

Frequently asked questions

Related scrapers

ChatGPT and other names and marks of chatgpt.com belong to their owner and are used here only to identify the website this API works with. ScrapeUnblocker is an independent service, not affiliated with, endorsed by or sponsored by the owner of chatgpt.com.