Swap Answer Order Before Trusting an AI Judge

Before trusting an AI judge to compare two answers, show it the same answers in both orders. Translate its choices back to answer identities, then check whether the winner changes. A judge that prefers whichever answer appears first can make a model upgrade look better simply because the evaluation harness always puts it first.

This is a practical extension of checking judge agreement. Agreement with reference labels and stability under reordered inputs answer different questions. The exercise below is a proposed synthetic test, not a report of production results or employer data.

Where the judge sits

A pairwise evaluation pipeline has four steps: collect two answers to the same question, place them into a judging prompt, request a verdict, and map that verdict back to the systems that produced the answers. The judge might return A, B, or tie. Those are presentation labels. They are not permanent model identities.

Position bias is a documented concern. The MT-Bench and Chatbot Arena paper examines position bias alongside verbosity and other limitations of model judges. A later systematic study of position bias distinguishes repetition stability, position consistency, and preference fairness. These findings motivate testing your own judge configuration; they do not establish its error rate.

For QA, the useful invariant is simple: moving an answer from the first slot to the second should not change its correctness. That gives us a transformation to test even before we have a large human-labeled dataset.

Build a pair with an obvious expected winner

Create a fictional support policy: unopened equipment can be returned within 14 calendar days. Ask how long a customer has to return unopened equipment. Answer X says 14 calendar days. Answer Y says 30 calendar days. Give both the same sentence structure so this first fixture isolates a factual difference.

Store X and Y as immutable answer IDs. Supply the policy as reference evidence, and instruct the judge to prioritize factual correctness over tone. Hide model names. In one request, place X in slot A and Y in slot B. In another, reverse their positions. Use separate requests with no shared conversation history.

The raw verdict should change here. The winning answer should not. Comparing the literal verdict strings would incorrectly flag the correct behavior as a failure.

Save the question, reference, answer hashes, slot mapping, rubric version, judge model identifier, available generation settings, request ID, raw response, parsed verdict, and normalized winner. Preserve ties, invalid responses, and transport failures as separate outcomes.

Test the evaluator plumbing first

Before spending model calls, replace the judge with deterministic stubs. One stub always selects slot A. Your test must flag a changed winner after reversal. A second stub selects the answer containing the fixture fact, regardless of slot. Both presentations must normalize to X.

Also feed the parser a tie, a malformed verdict, and an empty response. None should silently become a candidate win. Deliberately reverse the mapping in a local test version and require the known-winner assertion to fail. This negative control proves that your test can catch the exact wiring error it claims to detect.

A model judge can be unreliable, but a broken normalizer can manufacture the same symptom. Check the deterministic boundary before interpreting model behavior.

Separate order sensitivity from ordinary variation

One changed verdict does not prove systematic position bias. The judge might also disagree with itself when given the same order twice. Add repeated original-order and repeated reversed-order requests, using the same predefined budget for every case. Randomize execution order so all original presentations do not run before all reversed presentations.

For a small synthetic pilot, choose 20 answer pairs and three calls per presentation: 120 judge requests. These are exercise sizes, not a power calculation. Include clear winners, close comparisons, and identical-answer controls whose rubric requires a tie. Have a human establish expected outcomes for factual fixtures before examining judge results.

Report within-order disagreement separately from across-order disagreement. Track how often each answer wins in each position, and inspect whether the first or second slot is favored across cases. Repeated calls remain measurements within a case; they are not 120 independent examples. The earlier post on repeated evaluation runs explains that sampling distinction.

Keep the denominator visible

Suppose a synthetic report contains 20 pairs: 14 retain the same decisive winner across the selected two presentations, four switch winners, one produces a tie, and one has a parser failure. The illustrative switch fraction among pairs with two decisive verdicts is 4 / 18, or 22.2%. It is not a measured result from this exercise.

Show all four outcome counts beside that fraction. Dropping ties and parser failures without reporting them makes the judge look more dependable than the evidence permits. If using repeated calls, specify the aggregation rule before running them and retain the individual verdicts. Do not rerun only inconvenient comparisons until the preferred winner appears.

Turn the evidence into a release decision

My proposed starting gate is strict for plumbing: no incorrect identity mappings, no silently accepted invalid verdicts, and all deterministic controls passing. For model behavior, set acceptable instability and factual-error limits before seeing results, based on the decisions the evaluator will support. Route unresolved comparisons to review rather than counting them as wins.

Order consistency is necessary evidence for this comparison workflow, but it is not proof of quality. A judge can consistently prefer the wrong answer. Keep reference-based accuracy checks alongside the swap test, and rerun both when the judge, rubric, or prompt template changes.

If your evaluator says the candidate won, can you show that the same answer still wins from the other slot?