Does pinning the judge model version stop it from drifting?

Not fully. A pinned version string is a record, not a guarantee: a hosted provider can serve updated weights behind the exact same alias, and nothing in the identifier changes when it happens. What pinning buys is a field on every verdict you can group by afterward, turning an unanswerable retro into a query instead.

Researchers testing LLM-as-judge reliability ran 118 evaluation runs across five weeks in 2026, covering 21 judges from nine providers, and said plainly in their limitations that hosted endpoints are known to drift across provider-side updates, sometimes silently, and that they never re-tested the same judges months later to see whether any held steady. If a study built specifically to measure judge reliability won’t claim an endpoint held still, a pinned version string in your own pipeline doesn’t buy that guarantee either.

Recalibrate the judge whenever you have reason to think the endpoint moved, not only when you deliberately change the pinned version.

sources

LLM-as-judge reliability study across 21 judges and nine providers fetched 25 aug 26

keep reading

More on this.

Does running more LLM judges on the same trace reduce error?Not by much: a 2026 study found nine LLM judges carried only about two judges' worth of independent signal, since they fail on the same items.answer →How does an LLM judge compare to human evaluation?About as often as two human graders agree with each other: 85 percent versus 81 percent on MT-Bench, the benchmark that established LLM-as-judge as a working method.answer →Is 90 percent agreement good enough for an LLM judge?Not on its own. Raw agreement doesn't correct for chance, so on a skewed label set 90 percent can sit on a kappa near zero.answer →Everything on LLM as judgeThe definition, and every question answered under LLM as judge.concept →

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization

traces: 10,000 per calendar month
stored trace data: 1 GB
retention: 30 days
model credit: $10, one-time, for triage and root-cause analysis
credit card: not required

start for free

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

set up with a coding agentinstall manually

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y