← Home
Case study 2 of 3 · Email Triage · Built and presented to the company

Triaging sales email

I built a system that finds quote requests in a shared inbox, checks the part numbers, and drafts replies for the sales team to review.

The problem

A shared sales inbox mixes quote requests with vendor notices, invoices, newsletters and spam. The people answering it read everything to find the handful of emails that need a price. Part numbers get retyped by hand, and nobody has a count of which products customers ask about most.

Who needed it: the sales and support team. The demand count also matters to purchasing.

How it works

  1. Rules first Cheap deterministic rules catch the obvious: known senders, invoices, notifications.
  2. Classify What is left goes to a language model that sorts it into fourteen categories. Only sales mail stays; the rest goes to a "not sales" list a person can skim.
  3. Extract Part numbers come from catalog matching first, then a model for free-text requests.
  4. Check Every extracted part number is looked for in the email text. One that is not there is flagged as invented and held.
  5. Rank A queue orders what needs a reply, with catalog-priced draft replies.
  6. Count A demand command tallies requests by product over time.
  7. Review A person works the ranked queue. Nothing is sent automatically.

Decisions that shaped it

  • Rules first, model second. Plain rules settle what they can. The model only sees what is left, and a low-confidence answer goes to a person. Letting the model process everything was the obvious first design, and real mail killed it: the model kept mistaking first requests for follow-ups.
  • Nothing the model extracts is trusted until it is found in the email. Every part number is checked against the text the model was shown. Quote lines are built only from catalog matches, so an invented part can never be priced.
  • Read-only by construction. The mail client opens folders read-only and refuses write commands by name before they are sent. An independent review found a gap, and I closed it with an allowlist.
  • Email is hostile input. Hidden text is stripped and flagged as fraud, text written to steer a model is held for a person, and every email is wrapped in random boundaries before it reaches a model.
  • Every answer carries a version. Prompts, question sets and models are versioned, and the page reads exactly one version per stage, so results from different setups never mix.
  • Spend is capped twice. A per-run cap and a daily cap, backed by a ledger that counts failed calls too, so an unattended run cannot quietly burn money.
  • Cut scope hard. In one day I removed four department readers, three extraction routes and a model judge. A tool one person owns has to be readable in an afternoon.

The number that was 16 times too big

Two days before presenting, I audited my own slides against the code. The demand column was multiplying price by quantity when the form's price was already a line total: $2.3M on the slide against $146K in the test data. I fixed the code, fixed the slide, and kept the audit in the repository.

The same audit pointed out that accuracy on synthetic mail says little about real mail. So the next thing I built was a blind labeling quiz and a scoring harness, and I labeled real emails by hand to find out.

Results

3,300+emails processed. Messages are stored locally, and card and account numbers are masked before text is sent to the model
1,180+automated tests across 113 files
1.5 sfirst page load, down from about 50 seconds after I cached each reader's output
700synthetic emails with expected labels driving the offline tests

On real emails I labeled by hand, without seeing the system's answers:

QuestionSystem agreed with me
Is this a customer?22 of 24
Is this suspicious?23 of 27
Is anyone still waiting on us?6 of 6
What kind of email is it?18 of 29

The harness also showed exactly where the next work goes: urgency scoring and part extraction on real mail. The next change now has a number to beat.

My part

I was product owner, sole labeler and reviewer, directing AI coding agents: I defined the categories and the rubric, set the safety rules, commissioned independent reviews of the risky parts, and decided what to cut.

Intended design: actively reading and sorting emails as they arrive, and sending routine replies on its own once measured accuracy earns it. That one is deliberately last.

Stack
Python, pytest, SQLite, LLM APIs through HTTP and the Claude Code CLI
Repository
Private; fixtures are synthetic.