A LoRA fine-tune of Mistral 7B on 10,000 examples cost me under $50 on RunPod in 2026 — the compute is cheap and getting cheaper. The bottleneck moved years ago, and most teams haven’t noticed. It is no longer the training run. It is the quality of the data you feed into it, and annotation quality is where fine-tuning projects quietly fail.
What Annotation Quality Actually Means
Annotation quality is not just “did the annotators agree with each other.” It is three harder questions. Did the annotation rubric actually capture the task you care about, or a convenient proxy for it? Were annotators calibrated against a shared set of examples before they started labeling, or did they each invent their own interpretation? And were the edge cases documented as they came up, or silently resolved and forgotten? Agreement on a bad rubric just gives you consistent noise. The quality lives in the rubric and the calibration, not in the raw agreement number alone.
Inter-Rater Reliability as the Starting Gate
Cohen’s kappa is the first gate, and I treat it as a gate rather than a vanity metric. A kappa below 0.70 means your annotators are not reliably applying the same rubric — which means your fine-tuning data carries the disagreement as noise straight into the weights. A kappa below 0.60 tells you something more serious: the rubric itself probably needs redesign, not just more annotator calibration. The distinction matters because the fix is different. Below 0.70 you retrain the people. Below 0.60 you rewrite the instructions.
The internal tax scenario classifier Annotation Workflow
Building the internal tax scenario classifier meant creating more than 10,000 training examples for tax scenario classification, and the workflow reflected everything above. Three annotators reviewed each batch. Every batch opened with a pre-calibration session using 50 canonical examples, so everyone was anchored to the same interpretation before touching new data. A resolution protocol handled disagreements — an adjudicator reviewed any annotator pair that fell below a kappa of 0.75. The first 3,000 examples took six weeks. The annotating itself was fast; what was slow was resolving rubric ambiguities, the genuinely hard classification calls where reasonable experts disagreed and the instructions had to be sharpened before we could proceed. That slowness was the work paying off, not the work failing.
The Failure Mode
The common failure is predictable. A team rushes annotation, skips the pre-calibration session, skips the inter-rater checks, and ships a large pile of labeled examples on a deadline. Then they fine-tune and discover the model is worse than the base model on their own task. They blame the model, the framework, the learning rate — anything but the data. The model learned exactly what it was given, which was noise dressed up as signal. There is no hyperparameter that recovers from a corrupted training set.
The Quality Floor
Here is the number that should reshape how teams budget annotation effort: 1,000 high-quality annotated examples with a kappa above 0.80 beats 20,000 noisy ones for fine-tuning. This is consistent across the fine-tuning literature and consistent with my own production experience. The instinct to gather more data is usually the wrong instinct. The right instinct is to make the data you already have cleaner — tighten the rubric, recalibrate the annotators, and accept a smaller, sharper training set over a large, blurry one.
Documenting Edge Cases as Your Test Set
For every annotation batch, I require documenting the 20 to 30 cases that were hardest to classify and why they were hard. Those documented edge cases become the test set for evaluating the fine-tuned model — deliberately, not random samples pulled from the annotation pool. A random test set tells you how the model does on average. The hard-case test set tells you how it does exactly where it’s most likely to break, which is the only place where evaluation earns its keep. The same rigor shows up across regulated deployments, which I wrote about in lessons from 17 enterprise projects.
What Transfers
This is clinical assessment methodology applied to machine learning annotation. Structured clinical interviews, rubric-based scoring, inter-rater reliability checks — all of these exist in clinical psychology, and they exist there for exactly the same reasons they matter in annotation. When you need multiple humans to apply the same judgment consistently to ambiguous material, the discipline is identical whether the material is a patient presentation or a tax scenario. The field that has spent decades measuring human agreement has a great deal to teach the field that just discovered it needs to.
→ Building something at the intersection of AI, edge computing, and behavioral science? Let’s connect.