Tag: llm-evaluation

LLM-as-judge is seductive because it solves an expensive problem cheaply: instead of paying human annotators to evaluate model outputs, you have a larger or more capable model do the grading. The cost savings are real. The risks are also real, and teams that adopt LLM-as-judge without understanding the failure modes end up with evaluation infrastructure that gives them confident-sounding metrics while quietly measuring the wrong thing.

When It Works

LLM-as-judge works well when the evaluation criteria are explicit and the judge model can be given precise instructions that human annotators would also receive. Factual accuracy, format compliance, completeness against a checklist, tone matching against a defined style guide — these are tasks where a capable judge model can achieve inter-rater reliability with human annotation that makes it a reasonable substitute. The test for whether your LLM-as-judge setup is working is to run it in parallel with human annotation on a sample and calculate Cohen’s kappa between the two. If kappa is above 0.7, your automated evaluation is reliable enough to trust at scale. Below that threshold, you’re measuring something that doesn’t sufficiently correlate with human judgment to be useful.

When It Fails

LLM-as-judge fails predictably on tasks that require domain expertise the judge model doesn’t have, tasks where the rubric is genuinely ambiguous, and tasks where the judge model has systematic biases that don’t match the target population. On domain expertise: a general-purpose model grading the accuracy of specialized tax scenario classification outputs is grading against its own training data, not against ground truth. The grade it gives correlates with how confident the output sounds, not whether it’s correct. On rubric ambiguity: if the evaluation criteria aren’t precise enough for two human annotators to agree consistently, the judge model will also be inconsistent — but it will be inconsistent in a way that looks consistent because it applies the same internal model repeatedly. On bias: LLMs have documented preferences for certain output styles, lengths, and formats that don’t always correlate with quality. A judge model that prefers longer, more hedged responses will systematically grade those outputs higher regardless of their actual correctness.

Calibration Against Human Annotation

The calibration protocol that worked in the internal tax scenario classifier — the tax scenario processing system that ran 415,000 scenarios in 4 hours — applied the same methodology to LLM-as-judge validation. Before deploying automated evaluation at scale, we sampled 200 outputs and had both human annotators and the judge model evaluate them independently. The kappa between human annotators came in at 0.81 on unambiguous cases and 0.54 on boundary cases. The kappa between human annotation and LLM-as-judge came in at 0.78 on unambiguous cases and 0.41 on boundary cases. That calibration told us exactly where to trust the automated evaluation and where to require human review: for clear-cut cases at scale, the judge model was reliable; for boundary cases, it was not, and routing those to human annotators was the correct design.

The Consistency Trap

The most dangerous property of LLM-as-judge is that it can be consistently wrong. A human annotation process with poor inter-rater reliability at least has the property that its errors are somewhat random — with enough annotators, the noise averages out. A judge model applies the same systematic bias repeatedly, which means the errors don’t average out; they compound. If your judge model systematically misclassifies a category of outputs, every data point in that category gets the wrong grade, and your optimization process confidently drives the model you’re evaluating in the wrong direction. The consistency that makes LLM-as-judge seem reliable is also what makes its systematic errors harder to detect.

Practical Implementation Guidelines

Three guidelines have held up across my use of LLM-as-judge in production systems. First, calibrate before deploying: run the judge in parallel with human annotation on a representative sample and calculate kappa before trusting it at scale. Second, stratify by case type: most evaluation tasks have a subset of cases where automated grading is reliable and a subset where it isn’t — treat those subsets differently rather than applying a single threshold. Third, re-calibrate periodically: the judge model’s behavior can change across versions, and a calibration established on one model version doesn’t automatically transfer to the next.

What Transfers

The clinical assessment parallel holds here too. Structured clinical interviews have a validated set of questions and scoring criteria specifically because clinician judgment varies in ways that affect diagnostic reliability — the same symptoms get different interpretations from different practitioners without standardized tools. LLM-as-judge is automated judgment, and it has the same reliability problem as human judgment: it needs to be calibrated, validated, and monitored rather than trusted by default. The tools for doing that — inter-rater reliability measurement, calibration protocols, stratified validation — come from the same field that figured this out for human judgment decades ago.

→ Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.

I have watched teams ship LLM features with no structured evaluation whatsoever — no golden dataset, no inter-rater agreement calculation, no adversarial examples — and then express genuine surprise three weeks later when production breaks in ways that dev never predicted. This is not a tooling problem or a research problem; it’s a discipline problem, and the fix is unglamorous: you need a curated, annotated evaluation set built before you ship, not assembled in a panic after your first production incident.

What a Golden Dataset Actually Is

A golden dataset is a curated collection of (input, expected_output) pairs that represent the real distribution of tasks your model will encounter in production — including the hard ones, the edge cases, and the inputs specifically designed to surface failure modes. It is not a sample of the easy inputs where the model already performs well. It is not a set of examples you grabbed from a tutorial. It is a structured artifact, maintained like code, that tells you whether your model is getting better or worse as you iterate. Three ingredients separate a real golden dataset from a list of test cases someone threw together on a Friday afternoon. First, it needs at least 500 examples — below that threshold, your accuracy measurements have confidence intervals wide enough to be meaningless. Second, it needs adversarial examples mixed with the easy ones in a ratio that reflects your actual production failure rate — if 15% of your production inputs are edge cases, your golden set should be at least 15% adversarial. Third, every example needs to be annotated by at least two independent annotators, with inter-rater reliability calculated using Cohen’s kappa — and your target kappa should be 0.7 or higher before you trust the labels.

How the internal tax scenario classifier Forced the Discipline

When I built the internal tax scenario classifier — the tax scenario processing system that ran 415,000 scenarios in 4 hours and achieved a 31% reduction in purchase order error rate — the evaluation problem was not optional. Tax classification is high-stakes and adversarially complex: the same economic transaction can have materially different tax treatments depending on jurisdiction, contract structure, and a half-dozen other variables. Getting the model wrong didn’t produce a slightly inconvenient user experience; it produced incorrect tax filings with audit exposure attached. We built an annotation corpus of 8,000+ labeled tax scenarios before we ran a single evaluation pass. The annotation workflow ran in three phases: a senior tax specialist assigned the ground-truth label for each scenario, a second specialist reviewed independently, and disagreements went to a structured resolution process where both annotators discussed the case against a written decision criteria document before a label was assigned. The kappa score in the first annotation round was 0.61 — below the 0.7 threshold — which told us the labeling criteria were ambiguous in about 20% of the cases. We spent two weeks tightening the criteria document and re-annotating the contested examples. The second pass came in at 0.74. That kappa score meant something: when the model scored 88% accuracy against that dataset, we had high confidence the 88% was real and not an artifact of easy examples and loose labels.

The Distribution Trap

The most common mistake teams make after building a golden dataset is constructing one that is too easy — heavy on the clear-cut cases, light on the ambiguous ones — and then celebrating when the model hits 94% accuracy. That number is meaningless if your production inputs include the 6% of scenarios where the model catastrophically fails, because that 6% might represent the highest-stakes decisions in the entire workflow. The adversarial examples you add in week three of evaluation should not be random stress tests — they should be the actual failure modes you observed when you let the model run on unlabeled production data in shadow mode. Shadow mode evaluation — running the model in parallel with the existing process without using its outputs, just observing — is how you discover what your golden set is missing. The failure modes you find in shadow mode become your adversarial examples, which makes your golden set a living artifact that improves as you learn more about where the model breaks.

What Transfers

Clinical assessment methodology has required structured reliability measurement for decades — the DSM field trials that established diagnostic criteria used inter-rater kappa as a primary validity metric, and a diagnostic tool with kappa below 0.8 was considered insufficiently reliable for clinical use. That standard didn’t emerge because psychologists are unusually rigorous; it emerged because the cost of diagnostic error — misdiagnosis, missed treatment, wrong intervention — made reliability non-negotiable. LLM evaluation should be held to the same logic. If the model’s output touches a consequential decision, the annotation process that produces your ground truth needs enough rigor that two independent experts agree on the right answer at least 70% of the time after accounting for chance agreement. Below that threshold, you’re not measuring model performance — you’re measuring label noise. The investment in evaluation infrastructure is front-loaded and genuinely tedious, but it’s the only thing that gives you honest signal about whether your model is improving or whether you’re just iterating on vibes.

→ Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.