Does a judge that ranks well on one benchmark rank well on another?

Not reliably. The largest evaluation of LLM judges to date, 21 judges scored across roughly 541,000 judgments over 118 runs, found that judge rankings shift by up to 14 positions when the same judges are scored on a different benchmark. A judge that tops one leaderboard can land in the middle of the pack on another.

The paper’s own name for this is “reliability without validity”: a judge can be consistent within one benchmark and still give you no stable answer to which judge is actually best, because the ranking depends on which benchmark you picked to ask. A public leaderboard position is a weak reason to choose a judge for your own grading pipeline.

The only ranking that means anything for your use is the one you build against your own labeled traces, since a judge’s score on someone else’s benchmark doesn’t transfer to your rubric, your traces, or your definition of a correct answer.

sources

keep reading

More on this.

Do LLM judges score markdown-formatted answers higher than plain prose?Yes, and it's the largest bias measured: style bias runs 0.10 to 0.76 across judges, far past the 0.04 ceiling for favoring whichever answer came first.answer →Does a bigger model make a better judge?No. A 20-model study found self-preference bias uncorrelated with capability, and a separate 541,000-judgment study found scale doesn't stabilize rankings.answer →Does a more capable LLM judge show less self-preference bias?No. A 20-model study found more capable LLM judges show no less self-preference bias, sometimes more; a restructured scoring prompt cut it 31.5%.answer →Everything on judge bias researchThe definition, and every question answered under judge bias research.concept →

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization
traces
10,000 per calendar month
stored trace data
1 GB
retention
30 days
model credit
$10, one-time, for triage and root-cause analysis
credit card
not required
start for free

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y