Agent reliability

Graders

A grader is a check that reads a trace of an agent's behavior and returns a verdict about one specific thing the agent was supposed to do. Running one grader across many traces turns a general sense of quality into a rate for one behavior, and a rate can be compared across agent versions and over time.

Graders come in three forms, ordered by cost per verdict. Deterministic checks are code that inspects the trace, cheap enough to run on every output. Trained classifiers read text and score a single property at low cost. LLM judges use a model to reason about the output. They cost the most per verdict and cover judgments that require reasoning about meaning.

Two properties determine what a grader's verdict means. The first is scope. A grader tied to one observable behavior, with a stated condition for when it applies, gives a verdict that says exactly which behavior failed. As scope widens toward overall quality, each verdict carries less information about what actually happened. The second is origin. A grader written from a failure observed in real traffic measures something that actually occurs. A grader written from speculation measures what its author imagined, which may differ from what the agent does in practice.

Graders also get stale when the agent's intent changes. Each grader encodes what the agent was supposed to do when it was written, so when the intent changes, the grader keeps judging against the old one. It still returns verdicts, so its rate can move for reasons unrelated to what the agent is now meant to do.

5 questions

Answered, plainly.

Does a regression grader have to be an LLM judge?No. A deterministic check that asserts the exact condition that broke is cheaper and more stable than an LLM judge, which only earns its cost on failures that need judging meaning.answer →What stops a description from producing a grader that measures the wrong thing?Nothing by itself. A description only encodes what its author imagined the failure looks like, and needs grounding in a real trace where that failure actually happened before it ships.answer →How do you know when a grader no longer matches what the agent does?Not from the pass rate. Check whether the prompt, tool contract, or output schema the grader was written against has changed since, because it keeps judging the old version either way.answer →What does an eval score actually claim about my agent?It's a claim about the eval set the graders ran on, not the agent as a whole; it says nothing about a failure mode no grader in the set checks for.answer →What is a grader?A grader is a check that reads a trace of an agent's behavior and returns a verdict about one specific thing the agent was supposed to do.answer →

Send us the traces you already emit.

book a demo