The set is the specification
A team at Northwind Commodities spent three weeks tuning a prompt to pull counterparty, settlement date, quantity and incoterm out of physical trade confirmations. Every reviewer had a different sense of whether it was working, because every reviewer was pasting in different documents. The feature had no specification. It had opinions.
The fix was unglamorous: 200 real confirmations, labelled by two commodity operations staff, stratified so that roughly 60 per cent were the formats that make up the daily volume, 25 per cent were formats the desk already knew caused trouble, and 15 per cent were genuinely broken inputs — scanned at an angle, two contracts in one PDF, a fax header across the first line.
Labelling cost about four minutes per document, so thirteen hours of expert time in total. That is less than the cost of one rollback and considerably less than the cost of a wrong settlement date reaching a counterparty. The set became the thing the team argued about, which is the correct thing to argue about.
Score fields, not documents
A single accuracy figure is close to useless. On the first honest run, the Northwind extractor scored 91 per cent overall, which sounds shippable until you break it down: counterparty name 98 per cent, quantity 96, incoterm 94, settlement date 71. The settlement date was the field the downstream risk engine cared about most. Report per field or you are reporting nothing.
Weight the failure modes too. A missing value that routes to a human is an inconvenience; a confidently wrong value that flows into a position report is an incident. We score those separately and track abstention as its own outcome, because a model that says nothing when it is unsure is behaving correctly and should not be penalised as though it hallucinated.
Where a human label is expensive, a model judge is workable, but only after you have measured it. Compare the judge against 100 human-labelled cases and look for agreement above about 0.7 on Cohen's kappa before trusting it unsupervised, then re-audit ten per cent of its verdicts every month. An unaudited judge drifts quietly and takes your metrics with it.
Run it on every change, in CI
Prompts, model versions, chunk sizes, retriever settings and post-processing regexes are all code, and all of them move the numbers. The eval runs as a build step on every pull request touching any of them, with the model version pinned explicitly rather than tracking a floating alias. The merge gate is simple: no field may drop more than two points against the last release.
Keep the run cheap enough that nobody is tempted to skip it. Two hundred cases across three seeds is roughly four minutes of wall clock and between three and six pounds of inference for a mid-sized model. An evaluation that takes forty minutes and thirty pounds gets disabled in a hurry the week before a deadline.
Report the median and the spread across seeds. Temperature zero is not determinism, and teams routinely celebrate a one-point gain that sits well inside the noise band. If the spread across three seeds is 1.8 points, then a 1.2-point improvement is not an improvement.
The set decays, so grow it deliberately
Distribution shifts. A new counterparty starts sending confirmations from a different system, a regulation changes the wording of a clause, and a set built in March quietly stops representing August. We run a standing rule: every production failure becomes a fixture in the set within a week, with the offending input and the correct output attached to the ticket that reported it.
Hold back a slice — typically 20 per cent — that is never used for prompt tuning and is opened only at release candidates. Without it you will overfit the prompt to the evaluation and discover the gap in front of a customer. It happens more often than anyone admits, because the feedback loop from tuning against the whole set is fast and gratifying.
At Kestrel Industrial, the equivalent set grew from 180 cases to 640 over eight months of live running. That growth is not a sign the project went badly. It is the record of everything the team learned about its own inputs, and it is the asset that makes the next model upgrade a one-day job instead of a re-litigation.
What this changes with the business
The conversation with a sponsor stops being about impressions and starts being about a sentence like this: field-level accuracy of 94 per cent across 640 cases, 5 per cent of documents routed to a human reviewer, 1.9 pence of inference per document, and no field below 88. That sentence can be argued with, budgeted against and audited.
It also gives you the ability to say no honestly. On one engagement the numbers made clear that clause-level interpretation was going to sit around 80 per cent on the hard slice no matter how the prompt was written, which was not good enough for the intended use. We shipped extraction, dropped interpretation, and saved the client a quarter of wasted delivery. The evaluation set is what made that a five-minute decision rather than a five-month one.