Does a bigger model make a better judge?

No. A study measuring self-preference bias across 20 mainstream models found capability uncorrelated with bias, and in some cases negatively correlated, meaning a stronger judge model can favor its own outputs more than a weaker one does, not less. What actually cut the bias, by about 31.5% on average, was a structured, multi-dimension scoring prompt, a change to how the judge is asked to score, not a bigger model doing the scoring.

Scale doesn’t buy stability either. A separate study spanning 21 judges from nine providers and roughly 541,000 judgments found judge rankings shift by up to 14 places depending on which benchmark measures them, and that raw agreement with human labels overstates judge quality by 33 to 41 percentage points once corrected for chance agreement. Neither finding tracks model size. The fix in both studies is the same: calibrate the judge against your own labeled cases instead of trusting a bigger name on the model card, and structure the scoring prompt rather than just upgrading it.