Scraping product listings and extracting data with an LLM

Scraping product listings and extracting structured data from them is two problems wearing one name. The first is fetching pages without hurting the sites you depend on. The second is turning inconsistent, half-structured retailer pages into clean records your code can trust. An LLM helps a lot with the second problem and not at all with the first. This post covers both, from building RigRadar (which scrapes local hardware-retailer listings and uses an LLM to recommend gaming-PC builds) and from a separate scraper I run nightly.

Fetching politely is the real engineering

In practice the hard part is staying a well-behaved guest, because a scraper that gets blocked produces no data at all. These are the rules my nightly scraper runs by:

• Space requests to what the server tolerates. Start slow and tune the delay per site. A fixed, modest pace that never trips rate limits beats a fast one that does.
• Back off on 429 and 403. Both responses mean "slow down" in practice. Retry them with exponential backoff, and give up on that URL after a few attempts.
• Run a preflight check. Before a full run, fetch one known page per site and confirm it returns what you expect. If the preflight fails, skip the run and alert.
• Stop when you are refused. If requests are refused across several stores, the problem is on your side. My scraper pauses all requests for 15 minutes after a refusal, then asks again, and halts the run if it is still refused. Hammering a server that has already said no is how a temporary block becomes a permanent one.
• Respect platform page limits. If a storefront caps how many pages of results it serves, stop there.
• Ask for compressed responses. Sending an Accept-Encoding header and confirming the server honours it cut one page from about 905 KB to 87 KB. Across a night's run that took transfer from about 1.1 GB to around 120 MB. Less bandwidth on both ends.

I'd add one thing to that list that has nothing to do with HTTP: log every run and every fetch. My scraper writes each run and each request to run tables (URL, status, bytes, duration, outcome), and sends an email alert when a run is blocked or mostly fails. Without that history, "the data looks thin today" is impossible to debug.

If a site does not want to be scraped, the respectful answer is to stop, check its terms, and look for an official feed or API.

Use rules before you use a model

Once you have the HTML, resist the urge to send the whole page to an LLM. Most of what you need is already structured somewhere: JSON-LD product markup, Open Graph tags, a product type field, a tag list. Pull those out with ordinary parsing first. It is free and deterministic.

For classification, cheap signals voting together go a long way. On the nightly scraper, a product's category comes from three sources: the product title's main noun (the strongest signal, because "Oak Desk Lamp" is a lamp whatever the tags say), the store's product type, and its tags, counted by vote. Here is the shape of that in TypeScript:

type Signals = { title: string; productType?: string; tags: string[] };

export function voteCategory(
  s: Signals,
  lexicon: Record<string, string[]>
): string | null {
  const votes = new Map<string, number>();
  const score = (text: string | undefined, weight: number) => {
    if (!text) return;
    const t = text.toLowerCase();
    for (const [category, words] of Object.entries(lexicon)) {
      if (words.some((w) => t.includes(w))) {
        votes.set(category, (votes.get(category) ?? 0) + weight);
      }
    }
  };

  score(s.title, 3);
  score(s.productType, 2);
  for (const tag of s.tags) score(tag, 1);

  const ranked = [...votes.entries()].sort((a, b) => b[1] - a[1]);
  if (ranked.length === 0) return null;
  if (ranked.length > 1 && ranked[0][1] === ranked[1][1]) return null;
  return ranked[0][0];
}

When the vote returns null (no signal, or a tie), that listing goes to the LLM. Everything else never touches a model. The result is cheaper, faster and easier to explain than sending every item through a prompt.

Where the LLM earns its place

The model is the right tool for the long tail: listings where the structure is missing or inconsistent. Hardware listings are a good example. "RTX 4070 12GB OC White Edition (Open Box)" contains a GPU model, memory size, a condition and a colour, and every retailer formats it differently. Regex for every variant becomes a maintenance job of its own.

Here is how I'd approach the extraction step:

1. Send only the relevant text. Title, the price block, the spec table. Trim the rest. Smaller input is cheaper and gives the model less to get confused by.
2. Ask for JSON that matches a schema. Define the schema in Zod, describe it in the prompt, and validate the response. The full pattern is in getting reliable JSON from LLMs in TypeScript.
3. Allow null. Tell the model to return null for any field the text does not contain. A model told to always fill a field will invent a value.
4. Retry once, then park it. On a parse or validation failure, retry with the error message included. If it fails again, store the listing in a review queue.

import { z } from "zod";

export const Listing = z.object({
  category: z.enum(["cpu", "gpu", "motherboard", "ram", "storage", "psu", "case", "other"]),
  brand: z.string().nullable(),
  model: z.string().min(1),
  price: z.number().positive(),
  condition: z.enum(["new", "used", "open_box"]).nullable(),
});

export type Listing = z.infer<typeof Listing>;

Verify extracted values against the source

Schema validation proves the shape is right. It does not prove the values are true. For anything that matters, cross-check the model's output against the raw text:

• The extracted price should appear in the page's price block. If the model says 549 and the page never contains "549", reject it.
• The model name should be a substring (after normalising spaces and case) of the title or spec table.
• Categories should agree with any deterministic signal you do have. A listing whose product type says "Graphics Cards" should not come back as ram.

These checks are a few lines each and catch most hallucinated values before they reach a database. I'd keep them on even when the model looks reliable, because extraction sits upstream of everything else. On RigRadar, scraped listings feed the build recommendations, which only work if prices and compatibility data are right. That side is covered in building an LLM recommender that respects hard constraints.

Where I'd start

Get the fetching right first: slow, logged, compressed, with backoff and a hard stop when refused. Then extract with plain parsing and voting rules, and send only the leftovers to an LLM with a Zod schema and value checks against the source text. You can see RigRadar and the other things I've built on my projects page.