Why do judges flip their verdict when you swap the order of two answers?

Because the judge is weighing position along with content, a bias called position bias. The MT-Bench paper, which established LLM-as-judge as a working method, tested this directly: under the default prompt, GPT-4 kept its verdict only 65 percent of the time when the two answers it was comparing swapped places. The other 35 percent of the time, which answer won changed with nothing about either answer changing.

The same paper found GPT-4’s own outputs winning about 10 points more often than human votes gave them, and Claude-v1’s by about 25 points, though the authors are careful to say their data can’t establish self-preference as the cause.

None of this shows up in an overall agreement number, because these aren’t random errors, they track a condition: which model wrote the answer, or where it sat in the comparison. The fix is randomizing comparison order and grading each answer alone when you can, rather than trusting a single pairwise pass.

sources

keep reading

More on this.

Did my agent get worse, or did my judge?You can't tell from a falling score alone; run a fixed, human-labelled anchor set through the current judge on a steady interleave and watch it separately.answer →Does an LLM judge know when it's unsure?Somewhat: a judge's self-reported confidence predicts its own errors better than sampling it five times and voting, at a fifth of the inference cost.answer →Does pinning the judge model version stop it from drifting?Not fully. A pinned version string is a record, not a guarantee, since a hosted provider can serve updated weights under the same stable alias.answer →Everything on LLM as judgeThe definition, and every question answered under LLM as judge.concept →

Send us the traces you already emit.

book a demo