Does running more LLM judges on the same trace reduce error?

Not by much. A 2026 study called “Nine Judges, Two Effective Votes” found a panel of nine frontier LLM judges carried only about two judges’ worth of independent signal, since models trained similarly tend to fail on the same items. The panel’s accuracy landed 8 to 22 percentage points below nine truly independent votes, and the single best judge matched or beat the full panel.

The bottleneck is correlation between judges, not the aggregation method: models trained on overlapping data and instruction-tuned in similar ways tend to fail on the same inputs. Measuring a grader’s own accuracy still needs a labeled set to check against; stacking judges without that check just multiplies the cost of being wrong together.

sources

"Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels" (arXiv:2605.29800) fetched 09 sep 26

keep reading

More on this.

How does an LLM judge compare to human evaluation?About as often as two human graders agree with each other: 85 percent versus 81 percent on MT-Bench, the benchmark that established LLM-as-judge as a working method.answer →Is 90 percent agreement good enough for an LLM judge?Not on its own. Raw agreement doesn't correct for chance, so on a skewed label set 90 percent can sit on a kappa near zero.answer →Should an LLM judge return a pass/fail verdict or a score on a scale?Pass/fail, when the distinction is coarse enough for one. A finer scale needs quadratically more judge calls to trust at the same confidence.answer →Everything on LLM as judgeThe definition, and every question answered under LLM as judge.concept →

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization

traces: 10,000 per calendar month
stored trace data: 1 GB
retention: 30 days
model credit: $10, one-time, for triage and root-cause analysis
credit card: not required

start for free

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

set up with a coding agentinstall manually

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y