What does an eval score actually claim about my agent?

An eval score is a claim about the eval set, not about the agent. A high pass rate means most of the traces your graders ran on passed the specific checks those graders make, and nothing more. It says nothing about a failure mode none of your graders look for, and nothing about traffic your eval set doesn’t cover: a new user segment, a tool that just shipped, a prompt injection nobody’s tried yet.

Widening scope only helps up to a point. A grader tied to one specific behavior tells you exactly what broke when it fails. A single “overall quality” grader can pass while a real failure slips through underneath it, because meeting a broad standard says less about any one thing than a narrow check does.

Read a score as coverage of what you checked, not as a verdict on the agent.