90% Judge Agreement Can Miss Every Defect

A judge that agrees with reviewers on 90% of answers can still approve every defective answer in the test. Before using an LLM evaluator as a release gate, count which mistakes it misses. Agreement alone cannot tell you that.

Yesterday's golden-set exercise established the cases and expected behavior. This follow-up tests the evaluator against those expectations. Every count below is synthetic arithmetic for a QA exercise, not measured model performance or employer data.

Start with a deliberately bad judge

Imagine a support assistant answering questions from a fictional product manual. Reviewers inspect 100 saved answers against a frozen rubric: 90 satisfy the requirements and 10 invent an unsupported capability. Now imagine an evaluator that labels every answer pass.

Its agreement with the reviewers is 90 out of 100. Its ability to catch the ten known defects is zero. Both statements are true because the sample contains many more acceptable answers than defective ones.

Write the counts down before calculating a score:

The lower-left cell is the operational problem: ten false passes. Here, false-pass rate means false passes divided by reference failures, or 10/10. Define that denominator explicitly. Dividing by all 100 cases answers a different question and produces a much smaller-looking 10%.

What Cohen's kappa adds

Cohen's kappa measures agreement after accounting for agreement expected from the two label distributions. The scikit-learn documentation defines it as (observed agreement minus expected agreement) divided by (one minus expected agreement).

In this example, the reviewers pass 90% and fail 10%; the judge passes 100% and fails 0%. Expected agreement is therefore 0.9 times 1.0 plus 0.1 times 0.0, which equals 0.9. Observed agreement is also 0.9. Kappa is zero.

That exposes the weakness of this particular judge. It does not mean kappa is a universal release criterion. The statistic treats the labels symmetrically and does not encode the product cost of approving an unsupported capability. Keep the confusion matrix and the false-pass count beside it.

For a second synthetic judge, use counts of 88, 2, 4, and 6 in the same table order. Agreement rises to 94%. Expected agreement is 0.9 times 0.92 plus 0.1 times 0.08, or 0.836. Kappa is approximately 0.634. Yet four of ten defective answers still pass. An improved agreement statistic has not removed the missed-defect problem.

Build a reference reviewers can defend

For this exercise, I would ask two reviewers to label saved answers independently before seeing the automated verdict. Supply the same manual snapshot and a narrow rubric: required facts present, claims supported, and citations pointing to the supplied evidence.

Preserve initial disagreements. Resolve them by inspecting the requirement and source, and record the adjudication. If the source does not settle the question, retain an unresolved label instead of manufacturing certainty. Human review is a reference process that needs evidence; it is not automatically ground truth.

Keep unresolved cases out of the binary calculation and report their count separately. Apply the same discipline to evaluator abstentions and infrastructure failures. An unavailable judge must not silently become a pass, and dropping difficult cases must not silently improve the headline score. The earlier abstain-state post covers that decision boundary.

Separate defect discovery from traffic estimation

A synthetic challenge set should deliberately include failures: invented formats, missing conditions, stale document citations, and correct facts attributed to the wrong source. Its purpose is to expose weaknesses. Its defect frequency is chosen by the tester.

I would keep that set separate from a sample drawn from the intended workload. Report both, with their sampling rules. Do not pool them and call the resulting percentage a production defect rate. Adding twenty difficult examples changes the denominator as well as the task mix.

Ten reference failures are enough to demonstrate the arithmetic above, but they offer limited evidence about future behavior. Preserve the raw counts and avoid treating a small sample or one clean run as proof that the evaluator is dependable. Set acceptance criteria before examining candidate results, based on the consequences of each error type.

Test the judge when the answer has not changed

An evaluator can respond to presentation as well as correctness. The MT-Bench and Chatbot Arena paper documents limitations including position and verbosity bias. It motivates a useful QA check: change presentation while keeping the assessed facts fixed.

For pairwise comparisons, swap the two answer positions and map the verdict back to the original answer IDs. For individual grading, compare a concise answer with a longer paraphrase that preserves every relevant claim. Review the paraphrases first; accidentally changing meaning would invalidate the test.

Record verdict changes as evidence to investigate, not automatic proof of bias. Repeat a declared number of trials when outputs vary, and retain every result. A stable verdict is useful, but a consistently wrong verdict is still wrong.

Put the evidence next to the release decision

My proposed evaluator report would include the rubric and judge revisions, fixture IDs, reviewer adjudications, the confusion matrix, kappa, false passes by defect type, and unresolved or unavailable counts. Store the exact judge input and output so another engineer can reproduce the comparison as closely as the model service permits.

If a judge revision changes labels while the saved product answers remain fixed, investigate the evaluator before claiming that the product improved. For the next QA review, bring the ten answers the judge should reject. Whether it catches those defects is a more useful starting point than a lone agreement percentage.