Why do judges flip their verdict when you swap the order of two answers?

Because the judge is weighing position along with content, a bias called position bias. The MT-Bench paper, which established LLM-as-judge as a working method, tested this directly: under the default prompt, GPT-4 kept its verdict only 65 percent of the time when the two answers it was comparing swapped places. The other 35 percent of the time, which answer won changed with nothing about either answer changing.

The same paper found GPT-4’s own outputs winning about 10 points more often than human votes gave them, and Claude-v1’s by about 25 points, though the authors are careful to say their data can’t establish self-preference as the cause. Later work ties that bias to perplexity rather than self-recognition.

None of this shows up in an overall agreement number, because these aren’t random errors, they track a condition: which model wrote the answer, or where it sat in the comparison. The fix is randomizing comparison order and grading each answer alone when you can, rather than trusting a single pairwise pass.