Is 90 percent agreement good enough for an LLM judge?

Not on its own, because a raw percentage doesn’t subtract the agreement two raters would reach by guessing. Take a rubric where 90 percent of turns genuinely pass. Two raters who each say “pass” every single time agree on 90 percent of items and have learned nothing about the rubric; the chance-corrected kappa on those same labels is zero. The more skewed the label distribution, the further a raw percentage drifts from what it appears to say.

Use Cohen’s kappa if you have two raters, and Fleiss’ kappa or Krippendorff’s alpha for more than two; alpha also tolerates missing labels, which a raw percentage can’t handle at all. Any of these tells you something a target percentage can’t: how much of the apparent agreement is real signal rather than raters landing on the obvious majority answer.

The number alone still isn’t the whole check. Where the disagreements sit matters as much as how many there are: clustered on one edge case points to a rubric gap, while disagreement that tracks answer length or position is bias no rubric detail fixes.