LLM Evaluation at Scale: Building a Golden Dataset That Actually Works

I have watched teams ship LLM features with no structured evaluation whatsoever — no golden dataset, no inter-rater agreement calculation, no adversarial examples — and then express genuine surprise three weeks later when production breaks in ways that dev never predicted. This is not a tooling problem or a research problem; it’s a discipline problem, and the fix is unglamorous: you need a curated, annotated evaluation set built before you ship, not assembled in a panic after your first production incident.

What a Golden Dataset Actually Is

A golden dataset is a curated collection of (input, expected_output) pairs that represent the real distribution of tasks your model will encounter in production — including the hard ones, the edge cases, and the inputs specifically designed to surface failure modes. It is not a sample of the easy inputs where the model already performs well. It is not a set of examples you grabbed from a tutorial. It is a structured artifact, maintained like code, that tells you whether your model is getting better or worse as you iterate. Three ingredients separate a real golden dataset from a list of test cases someone threw together on a Friday afternoon. First, it needs at least 500 examples — below that threshold, your accuracy measurements have confidence intervals wide enough to be meaningless. Second, it needs adversarial examples mixed with the easy ones in a ratio that reflects your actual production failure rate — if 15% of your production inputs are edge cases, your golden set should be at least 15% adversarial. Third, every example needs to be annotated by at least two independent annotators, with inter-rater reliability calculated using Cohen’s kappa — and your target kappa should be 0.7 or higher before you trust the labels.

How the internal tax scenario classifier Forced the Discipline

When I built the internal tax scenario classifier — the tax scenario processing system that ran 415,000 scenarios in 4 hours and achieved a 31% reduction in purchase order error rate — the evaluation problem was not optional. Tax classification is high-stakes and adversarially complex: the same economic transaction can have materially different tax treatments depending on jurisdiction, contract structure, and a half-dozen other variables. Getting the model wrong didn’t produce a slightly inconvenient user experience; it produced incorrect tax filings with audit exposure attached. We built an annotation corpus of 8,000+ labeled tax scenarios before we ran a single evaluation pass. The annotation workflow ran in three phases: a senior tax specialist assigned the ground-truth label for each scenario, a second specialist reviewed independently, and disagreements went to a structured resolution process where both annotators discussed the case against a written decision criteria document before a label was assigned. The kappa score in the first annotation round was 0.61 — below the 0.7 threshold — which told us the labeling criteria were ambiguous in about 20% of the cases. We spent two weeks tightening the criteria document and re-annotating the contested examples. The second pass came in at 0.74. That kappa score meant something: when the model scored 88% accuracy against that dataset, we had high confidence the 88% was real and not an artifact of easy examples and loose labels.

The Distribution Trap

The most common mistake teams make after building a golden dataset is constructing one that is too easy — heavy on the clear-cut cases, light on the ambiguous ones — and then celebrating when the model hits 94% accuracy. That number is meaningless if your production inputs include the 6% of scenarios where the model catastrophically fails, because that 6% might represent the highest-stakes decisions in the entire workflow. The adversarial examples you add in week three of evaluation should not be random stress tests — they should be the actual failure modes you observed when you let the model run on unlabeled production data in shadow mode. Shadow mode evaluation — running the model in parallel with the existing process without using its outputs, just observing — is how you discover what your golden set is missing. The failure modes you find in shadow mode become your adversarial examples, which makes your golden set a living artifact that improves as you learn more about where the model breaks.

What Transfers

Clinical assessment methodology has required structured reliability measurement for decades — the DSM field trials that established diagnostic criteria used inter-rater kappa as a primary validity metric, and a diagnostic tool with kappa below 0.8 was considered insufficiently reliable for clinical use. That standard didn’t emerge because psychologists are unusually rigorous; it emerged because the cost of diagnostic error — misdiagnosis, missed treatment, wrong intervention — made reliability non-negotiable. LLM evaluation should be held to the same logic. If the model’s output touches a consequential decision, the annotation process that produces your ground truth needs enough rigor that two independent experts agree on the right answer at least 70% of the time after accounting for chance agreement. Below that threshold, you’re not measuring model performance — you’re measuring label noise. The investment in evaluation infrastructure is front-loaded and genuinely tedious, but it’s the only thing that gives you honest signal about whether your model is improving or whether you’re just iterating on vibes.

Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

Comments

Leave a Reply

Discover more from Grounded Intelligences

Subscribe now to keep reading and get access to the full archive.

Continue reading