Does an LLM judge score its own model's outputs more favorably?

Often, yes, though not for the reason it sounds like. A study measuring self-preference bias in LLM judges found GPT-4 rated outputs more favorably the more they resembled its own likely output, and traced the effect to perplexity rather than genuine self-recognition: judges score text with lower perplexity, text that reads like something the model itself would generate, higher than human evaluators do, whether or not that model actually wrote it.

That matters whenever a judge shares a model family with the agent it’s grading. A shared training distribution means shared phrasing habits, so a judge scoring by familiarity tends to reward outputs that sound like its own family’s voice over ones that don’t, independent of which answer is actually better. The guard is the same one any judge needs: calibrate it against human-labeled cases instead of trusting its verdicts on faith, and watch for a judge and its agent moving in the same direction together after a shared model upgrade.