How do I tell whether my judge is biased?

Swap the two things your judge is comparing, run both orders, and check how far its rate of picking “A” strays from 50%. A study across 21 judges and roughly 541,000 judgments found some judges hold near-perfect run-to-run consistency, above 0.95, while showing severe position bias, above 0.10, at the same time, because repeating your own answer and picking a position fairly are mathematically separate properties. A judge that always prefers whichever answer comes first will replicate that preference reliably, so test-retest consistency alone won’t catch it; only a paired swap will.

Run the swap at temperature 0 with caching off, across at least three repeats, and on more than one dataset: the same study found judge rankings moved by up to 14 places between benchmarks that measured different things, so a judge that looks clean on one label set can still be biased on yours.