The week the prices came back 83 times too high
In July I noticed one vendor's prices arriving about 83 times too high. The first investigation blamed the vendor's network, and a guard went in to quarantine jumps of that size. Six days later it happened again at 4.7 times, small enough to slip under the guard.
I bisected live requests until the cause was undeniable: my own tool's language header was making stores answer in a foreign currency. The fix pinned the market on every request, added a check that trips on currency evidence, and repaired about 26,000 rows from price history.
Since then, no guard goes in until it has caught real bad data and let real good data through.
The problem
Pricing research at a distributor used to mean opening vendor sites one at a time and copying numbers into a spreadsheet. It was slow, it went stale the day it was finished, and nobody could tell afterwards where a number had come from. The same part also appears under slightly different part numbers across vendors, so a naive search either misses matches or returns the wrong product.
Who needed it: the people who set prices and answer quotes, who need to see a competitor's number and the page behind it.
How it works
- Ingest Read the part list, normalize each part number.
- Discover Find candidate pages through vendor catalogs, search engines and site-specific adapters.
- Fetch Pull pages through HTTP, with a browser-automation rescue pass for sites that need it.
- Extract Pull title, price, availability and identifiers from each page.
- Match Decide whether the page is really the part. This is the hard part.
- Score Rank and grade the evidence per part.
- Store Write to SQLite with the source link kept on every row.
- Review A person reads the output, with the match reason shown next to each price.
Seven transform stages, each resumable with its own state, so a failed run picks up where it stopped. Data sources include platform adapters for Shopify, WooCommerce and Magento stores, sitemaps and price lists, plus five search engines.
Decisions that made it trustworthy
- Reject on meaning, not on keywords. A listing is thrown out as an accessory only when the part number is the object of "for", "fits" or "replaces". A keyword blocklist was the obvious route and I dropped it, because it rejected real products: "Manual Pull Station" contains the word manual.
- Flag it, never delete it. A suspicious price keeps its row, gets a named reason and leaves the statistics. Nothing disappears silently, so I can always see what the system decided and overrule it.
- Money is never guessed. An unknown currency is never assumed to be dollars, and a bare number is never divided by 100 on a hunch. Every price is stored in whole minor units with its currency.
- A 200-line ledger instead of a workflow platform. The job did not need a workflow platform. A small SQLite step ledger makes every step resumable: a run killed at part 31 of 100 picked up exactly where it stopped, with no repeated work.
- Curated catalogs first, web search second. Web search alone gave a usable price for about one part in eight. Moving to a reviewed list of vendor catalogs, with search filling the gaps, is what made the coverage real.
- A gate that cannot be talked past. Around 650 tests run before any change lands. A skipped check counts as a failure, every bundle has a minimum pass count, and the benchmark data is pinned by checksum.
Results
The benchmark grades each part as useful, thin or missed, separates parts that have no public price anywhere, and reports confidence intervals. It runs after every matching change, so a rule that recovers ten matches and breaks twelve shows up as a loss on the same day.
Failure handling
- A vendor that blocks or changes layout fails that stage for that host; the run continues and the host is reported.
- A match the rules cannot explain is stored as a candidate, not a price.
- Live runs are operator-gated. The pipeline never runs against vendor sites unattended.
My part
I direct coding agents to write the code. I set the product limits and the acceptance bars, design the benchmark and the trust rules, make the call when results disagree, and review what ships.
In the rebuild: showing stock per vendor, in stock, low or backorder, with how old the reading is. Intended design: sweeping the market on a schedule, so the picture stays current without anyone starting it.
- Stack
- Node.js, SQLite, node:test, YAML config, headless browser automation
- Repository
- Private.