← Home
Case study 1 of 3 · Market Price Pipeline · In use for pricing research

Automating price research

I used to check vendor sites one part at a time. Now I start a run with a list of part numbers and review the prices alongside their source pages.

One run covers hundreds of parts, with every price linked to the page it came from.

The week the prices came back 83 times too high

In July I noticed one vendor's prices arriving about 83 times too high. The first investigation blamed the vendor's network, and a guard went in to quarantine jumps of that size. Six days later it happened again at 4.7 times, small enough to slip under the guard.

I bisected live requests until the cause was undeniable: my own tool's language header was making stores answer in a foreign currency. The fix pinned the market on every request, added a check that trips on currency evidence, and repaired about 26,000 rows from price history.

Since then, no guard goes in until it has caught real bad data and let real good data through.

The problem

Pricing research at a distributor used to mean opening vendor sites one at a time and copying numbers into a spreadsheet. It was slow, it went stale the day it was finished, and nobody could tell afterwards where a number had come from. The same part also appears under slightly different part numbers across vendors, so a naive search either misses matches or returns the wrong product.

Who needed it: the people who set prices and answer quotes, who need to see a competitor's number and the page behind it.

How it works

  1. Ingest Read the part list, normalize each part number.
  2. Discover Find candidate pages through vendor catalogs, search engines and site-specific adapters.
  3. Fetch Pull pages through HTTP, with a browser-automation rescue pass for sites that need it.
  4. Extract Pull title, price, availability and identifiers from each page.
  5. Match Decide whether the page is really the part. This is the hard part.
  6. Score Rank and grade the evidence per part.
  7. Store Write to SQLite with the source link kept on every row.
  8. Review A person reads the output, with the match reason shown next to each price.

Seven transform stages, each resumable with its own state, so a failed run picks up where it stopped. Data sources include platform adapters for Shopify, WooCommerce and Magento stores, sitemaps and price lists, plus five search engines.

Decisions that made it trustworthy

  • Reject on meaning, not on keywords. A listing is thrown out as an accessory only when the part number is the object of "for", "fits" or "replaces". A keyword blocklist was the obvious route and I dropped it, because it rejected real products: "Manual Pull Station" contains the word manual.
  • Flag it, never delete it. A suspicious price keeps its row, gets a named reason and leaves the statistics. Nothing disappears silently, so I can always see what the system decided and overrule it.
  • Money is never guessed. An unknown currency is never assumed to be dollars, and a bare number is never divided by 100 on a hunch. Every price is stored in whole minor units with its currency.
  • A 200-line ledger instead of a workflow platform. The job did not need a workflow platform. A small SQLite step ledger makes every step resumable: a run killed at part 31 of 100 picked up exactly where it stopped, with no repeated work.
  • Curated catalogs first, web search second. Web search alone gave a usable price for about one part in eight. Moving to a reviewed list of vendor catalogs, with search filling the gaps, is what made the coverage real.
  • A gate that cannot be talked past. Around 650 tests run before any change lands. A skipped check counts as a failure, every bundle has a minimum pass count, and the benchmark data is pinned by checksum.

Results

75.8%of a fixed 100-part benchmark priced with a usable, source-linked price: 75 of the 99 parts that have a public price. Up from 70% in the first version
1.00match precision on 122 vendor pairs, each labeled twice by independent AI agents
128kcatalog rows from 26 vendor sources in the working database
650+automated tests that must pass before any change lands

The benchmark grades each part as useful, thin or missed, separates parts that have no public price anywhere, and reports confidence intervals. It runs after every matching change, so a rule that recovers ten matches and breaks twelve shows up as a loss on the same day.

Failure handling

  • A vendor that blocks or changes layout fails that stage for that host; the run continues and the host is reported.
  • A match the rules cannot explain is stored as a candidate, not a price.
  • Live runs are operator-gated. The pipeline never runs against vendor sites unattended.

My part

I direct coding agents to write the code. I set the product limits and the acceptance bars, design the benchmark and the trust rules, make the call when results disagree, and review what ships.

In the rebuild: showing stock per vendor, in stock, low or backorder, with how old the reading is. Intended design: sweeping the market on a schedule, so the picture stays current without anyone starting it.

Stack
Node.js, SQLite, node:test, YAML config, headless browser automation
Repository
Private.