Skip to content
rondahan.
All work
2026

ScrapeGoat

Live price comparison across four retailers that actively block scrapers

4
fallback extraction stages
4
retailers supported
90s
slowest scrape, run as a background job
Try the interface (opens in a new tab)

recorded run, not a live scrape

Repo (opens in a new tab)

What it does

Type a product name and ScrapeGoat searches Amazon, Best Buy, Walmart and Newegg at the same time, then shows the price, rating and review count from each one side by side. Everything is fetched live at the moment you ask, not read out of a cached database. Every result also carries the method that produced it and a trust score, so a solid number is distinguishable from a shaky one.

How it works

  1. Submit a job, do not block

    A full run takes 30 to 90 seconds, longer than a browser will hold a request open. The FastAPI layer creates a job, returns an ID straight away, and the frontend polls it.

  2. Fan out across retailers

    The orchestrator runs all four sites in parallel threads rather than in sequence, so total time is the slowest site instead of the sum of all four.

  3. Extract with a four-stage fallback

    Per site, strategies fire cheapest first and each only runs if the previous returned nothing: httpx + BeautifulSoup, then Playwright, then LLM text extraction, then Firecrawl.

  4. Match the right product

    rapidfuzz shortlists candidates by title similarity, then a second LLM pass checks the candidate really is the same product and rejects cases, cables and bundles.

  5. Score every result

    Each price records which stage produced it plus a trust score, so a wrong number can be traced to a stage from the payload instead of from logs.

  6. Stream progress back

    The Next.js UI polls the job and renders a live timeline of what each site is doing, so a 60 second wait reads as working rather than broken.

The shape of it

  1. Search query

    CLI or web UI

  2. in parallel:

    Amazon

    Best Buy

    Walmart

    Newegg

  3. per site, cheapest first:

    httpx + BeautifulSoup

    static HTML

  4. if empty:

    Playwright

    real browser

  5. if empty:

    LLM extraction

    from visible text

  6. if empty:

    Firecrawl

    managed API

  7. then:

    rapidfuzz

    shortlist by title

    LLM verify

    reject wrong product

  8. Price + method + trust score

Each extraction stage only runs if the one above it came back empty, so a cheap fetch handles most pages and the expensive stages stay rare. Every result records which stage produced it.

Tech choices

The decisions that shaped it, and why they went that way.

Playwright, because HTTP is not enough
Plenty of retail pages return 200 with no price in the HTML, because the price is injected by JavaScript a moment later. A real browser sees what a user sees.
The LLM is a fallback, not the default
Model calls are the slowest and most expensive stage, so they only fire when parsing has already failed. Most requests never reach them.
Anti-bot handling
Rotating browser profiles, simulated mouse movement and scroll with human timing, and page-settle heuristics that wait for the DOM to stop changing instead of for a fixed timeout. Scrapling covers fetcher-level fingerprinting.
Fuzzy matching alone was not safe
Title similarity confidently returned the wrong product often enough that it could not ship on its own, which is why the verification pass exists.

Built with

Backend
PythonFastAPIPlaywrightScraplinghttpxBeautifulSouppydantic
Frontend
Next.jsTypeScriptReact
Services
OpenAI APIFirecrawl
Matching
rapidfuzzLLM verification pass

What it produces

ScrapeGoat's search screen: a terminal-styled prompt reading 'what are you comparing?' above a query field and a scan button
The entry point. One query fans out to Amazon, Best Buy, Walmart and Newegg at once, so the wait is the slowest retailer rather than the sum of all four.
A run in progress at 62%: a status line reading 'Finished with Best Buy - no price found this time', a progress bar, a list of completed stores, and a small dodge game filling the wait
A live run. Because a full scrape takes 30 to 90 seconds, the API returns a job ID immediately and the UI polls it: the bar, the per-store line and the result cards all update as each retailer finishes. The dodge game fills the wait, which is the difference between a page that looks busy and one that looks broken.

How the repo is laid out

Roughly the order you'd read it in: entry point first, then the parts doing the work.

  • main.pyCLI entry: takes a query, prints the comparison
  • orchestrator.pyRuns all four retailers in parallel
  • site_runner.pyPer site: search, pick best match, extract
  • sites/One HTML adapter per retailer
  • extraction/The four-stage fallback and trust scoring
  • matching/Fuzzy title scoring; rejects bundles and accessories
  • api/FastAPI job server with live progress polling
  • web/Next.js UI: results, timeline, trust badges

Where it falls short

No automated test suite. The parts most worth testing are the ones that talk to live websites, and I had not found a mocking strategy I trusted. Running it end to end needs OpenAI and Firecrawl API keys.