What does an eval score actually claim about my agent?

An eval score is a claim about the eval set, not about the agent. A high pass rate means most of the traces your graders ran on passed the specific checks those graders make, and nothing more. It says nothing about a failure mode none of your graders look for, and nothing about traffic your eval set doesn’t cover: a new user segment, a tool that just shipped, a prompt injection nobody’s tried yet.

Widening scope only helps up to a point. A grader tied to one specific behavior tells you exactly what broke when it fails. A single “overall quality” grader can pass while a real failure slips through underneath it, because meeting a broad standard says less about any one thing than a narrow check does.

Read a score as coverage of what you checked, not as a verdict on the agent.

keep reading

More on this.

Does a regression grader have to be an LLM judge?No. A deterministic check that asserts the exact condition that broke is cheaper and more stable than an LLM judge, which only earns its cost on failures that need judging meaning.answer →What stops a description from producing a grader that measures the wrong thing?Nothing by itself. A description only encodes what its author imagined the failure looks like, and needs grounding in a real trace where that failure actually happened before it ships.answer →How do you know when a grader no longer matches what the agent does?Not from the pass rate. Check whether the prompt, tool contract, or output schema the grader was written against has changed since, because it keeps judging the old version either way.answer →Everything on gradersThe definition, and every question answered under graders.concept →

Send us the traces you already emit.

book a demo