Should an LLM judge return a pass/fail verdict or a score on a scale?

Pass/fail, when the distinction you need is coarse enough for one. How many judge calls it takes to trust a verdict scales with the square of how many levels the scale has: one analysis of judge sampling put a 0-1 scale with 2 bins, a pass/fail check, at needing about 6 calls to reach a given confidence, and the same scale with 4 bins needing about 36 calls for that same confidence. A finer scale asks the judge to draw more boundaries between levels at once, and each extra boundary needs its own evidence before you can trust it.

That cost is worth paying only when the decision genuinely needs the resolution, comparing quality across many runs on an ordinal axis, say, rather than a single per-trace verdict. For most graders, a few pass/fail questions, did it cite the source, did it follow the policy, give a clearer and cheaper signal than one number on a wide scale that everyone downstream still has to interpret.