# How do I tell whether my judge is biased?

Swap the two things your judge is comparing, run both orders, and check how far its rate of picking "A" strays from 50%. A study across 21 judges and roughly 541,000 judgments found some judges hold near-perfect run-to-run consistency, above 0.95, while showing severe position bias, above 0.10, at the same time, because repeating your own answer and picking a position fairly are mathematically separate properties. A judge that always prefers whichever answer comes first will replicate that preference reliably, so [test-retest consistency alone won't catch it](/answers/llm-as-judge/is-90-percent-agreement-good-enough); only a paired swap will.

Run the swap at temperature 0 with caching off, across at least three repeats, and on more than one dataset: the same study found judge rankings moved by up to 14 places between benchmarks that measured different things, so a judge that looks clean on one label set can still be biased on yours.

---

Sources:
- Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias (arXiv:2606.19544): https://arxiv.org/abs/2606.19544 (fetched 2026-09-24)

Source: https://tessary.ai/answers/judge-bias-research/how-do-i-tell-whether-my-judge-is-biased
More on Judge bias research: https://tessary.ai/answers/judge-bias-research
From Tessary, agent reliability for AI agents in production: https://tessary.ai