What is a grader in LLM agent evaluation?

A grader is a check that scores one trace against one specific failure mode, such as whether a tool call used the right arguments or whether a summary kept a required field. It is narrower than a generic quality score: a grader built for one failure mode should say nothing about a different one. Most agents need several graders, each covering a distinct way the agent can go wrong.

Should I use an LLM judge or a deterministic check for my agent?

Use a deterministic check whenever the answer is verifiable: did the tool call return a 200, does the output match a schema, is the required field present. Reserve an LLM judge for the cases that need semantic judgment, like whether an explanation is accurate or a response stays on topic. Route each failure mode to the cheaper check first, and save the LLM judge for what a rule genuinely can't answer.

How many examples do I need to calibrate an LLM-as-judge grader?

Enough that agreement stops moving when you add more, typically a few dozen labeled traces per failure mode, drawn from real production traffic rather than synthetic examples. Include cases the rubric author would disagree with a second reviewer about; those are the ones that reveal whether the rubric itself is ambiguous.

What's the fastest way to learn how to write LLM agent graders?

Start by reading ten to twenty of your agent's actual production traces by hand before writing any rubric. The failure modes worth grading are the ones you find there, not the ones a generic evaluation template assumes. Write the rubric from what you saw, then calibrate it against a second person's read of the same traces.

What agreement rate should I target between a grader and human labels?

Treat anything below roughly 80 percent agreement as a signal the rubric is ambiguous, not that the judge model is weak. Published LLM-as-judge work on general chat quality reaches into the mid-80s against human preference, which is close to the rate humans reach with each other on the same task. A grader built for one narrow failure mode, with a tight rubric, should be able to match or beat that.

Can the same grader work across different agents?

Rarely without changes. A grader encodes a rubric written against one agent's specific inputs, tools, and definition of correct behavior. Moving it to a different agent means re-checking whether the pass/fail examples still apply, since the failure modes that matter for a support agent are not the ones that matter for a coding agent.

How to Write LLM Agent Graders

The most cited LLM-as-judge result is still Zheng et al.’s MT-Bench and Chatbot Arena study: GPT-4 grading open-ended chat responses agreed with human preference about 85 percent of the time, ahead of the roughly 81 percent humans reach with each other. That number describes a general-purpose judge on general chat quality, not a judge built to check whether your refund tool used the right order ID.

Most teams that go looking for how to write LLM agent graders are really asking why their first attempt disagreed with a human reviewer on the same trace. The rubric, not the judge model, is usually the reason.

How to Write LLM Agent Graders in Practice

The process has four parts: write the rubric before you touch a prompt, decide whether the check needs an LLM judge or a deterministic rule, calibrate the judge against human-labeled traces, and only then point it at production traffic. Skipping the first two steps is why the fourth one produces a grader nobody trusts.

Start With the Rubric, Not the Prompt

Hamel Husain’s evals FAQ puts the rubric before the prompt for a specific reason: a judge prompt written against a vague rubric just moves the ambiguity into the model’s head, where you can’t inspect it. Write the rubric as an explicit pass/fail definition with a positive example, a negative example, and at least one edge case that a reasonable person could argue either way.

Then hand that rubric to a second person and have them label the same set of traces independently, before the LLM judge is involved at all. Where the two humans disagree, the rubric is the thing to fix, not either person’s judgment. Only after two humans converge on most of a labeled set does it make sense to ask an LLM to apply the same rubric.

LLM Judge or Deterministic Check: How to Decide

A deterministic check beats an LLM judge whenever the answer is verifiable: did the tool call return a valid response, does the output parse against a schema, is a required field present. These have one correct answer, and a regex or a type check grades them faster and more consistently than a model call ever will.

An LLM judge earns its cost on what’s left over: does the explanation match what actually happened in the trace, does the response stay on the topic the user asked about, is the tone appropriate for the situation. Route each failure mode to the cheaper check first, and only build an LLM judge for what a rule genuinely can’t answer.

Calibrate the Grader Against Human Labels Before You Trust It

A grader that hasn’t been checked against a human on the same traces is a guess with a confidence score attached. Take the labeled set from the rubric step, run the LLM judge against it, and compare verdicts row by row. Hamel’s evals FAQ frames the bar as a held-out test, not a fixed percentage: check the judge’s true positive and true negative rates against the labeled set, and if alignment is weak, try a different judge model before assuming the rubric itself is fine. A rubric that still can’t clear that bar after a model swap usually has ambiguous cases left to fix, not a model problem to solve with a bigger judge.

Where the judge and the human disagree, read the specific trace. It usually falls into one of the edge cases the rubric never described, which means the rubric gets a new example and the calibration set gets re-run.

Test the Grader Against Production Traces, Not Just Your Curated Set

A rubric calibrated against a curated set of clean examples will meet messier traffic the day it goes live: longer sessions, tool calls that partially succeeded, inputs nobody wrote a test case for. If your traces already sit in Langfuse, pulling a slice from there to test the calibrated grader against is faster than re-exporting a new sample by hand. Langfuse will track the score the grader produces over time. What it won’t tell you is which change moved that score, a prompt edit, a tool update, or a shift in an upstream agent’s output that never touched your repo at all.

That’s the gap a well-written grader alone doesn’t close. Even a grader that clears the calibration bar from the previous step will still misfire at some steady background rate, so a single low score on one trace is not evidence of anything. Reading the trend across many verdicts is what turns a misfire-prone grader into a reliable signal: a rate that holds steady at 3 percent and then climbs to 11 percent over a week is a real change, whether or not any single verdict in that window is one you’d personally agree with.

This is also where the rubric-writing work in this post turns into something a team doesn’t have to keep doing by hand. Tessary drafts a first version of the grader from your agent’s own observed traces, the same way you’d draft a rubric from reading twenty real examples, and the owner corrects it the way you’d correct a new rubric against a second reviewer’s labels. Once it’s running, Tessary watches that trend on live traffic, and when it breaks, works backward from the drop to name the specific change responsible, whether that’s sitting in your repo or came from a model provider update no diff would ever show. Task completion grading is one example of a grader built this way; how evals evolve past the first version covers what changes once real traffic starts correcting the rubric.

Get your first calibrated grader running against a real production trace, not a synthetic test set: connect a Langfuse source or your own OTLP traces and see where the current rubric disagrees with what actually happened, no credit card required.

Find what broke your agent