Does pinning the judge model version stop it from drifting?

Not fully. A pinned version string is a record, not a guarantee: a hosted provider can serve updated weights behind the exact same alias, and nothing in the identifier changes when it happens. What pinning buys is a field on every verdict you can group by afterward, turning an unanswerable retro into a query instead.

Researchers testing LLM-as-judge reliability ran 118 evaluation runs across five weeks in 2026, covering 21 judges from nine providers, and said plainly in their limitations that hosted endpoints are known to drift across provider-side updates, sometimes silently, and that they never re-tested the same judges months later to see whether any held steady. If a study built specifically to measure judge reliability won’t claim an endpoint held still, a pinned version string in your own pipeline doesn’t buy that guarantee either.

Recalibrate the judge whenever you have reason to think the endpoint moved, not only when you deliberately change the pinned version.

sources

keep reading

More on this.

Did my agent get worse, or did my judge?You can't tell from a falling score alone; run a fixed, human-labelled anchor set through the current judge on a steady interleave and watch it separately.answer →Does an LLM judge know when it's unsure?Somewhat: a judge's self-reported confidence predicts its own errors better than sampling it five times and voting, at a fifth of the inference cost.answer →Is 90 percent agreement good enough for an LLM judge?Not on its own. Raw agreement doesn't correct for chance, so on a skewed label set 90 percent can sit on a kappa near zero.answer →Everything on LLM as judgeThe definition, and every question answered under LLM as judge.concept →

Send us the traces you already emit.

book a demo