Where do the cases in an agent eval dataset come from?

Cases come from two places. Golden cases are hand-built: someone who understands the agent writes an input and the behavior expected on it, curated and reviewable, which is why they double as the ground truth a grader gets calibrated against. Production traces are the other source: real sessions the agent already handled, messy inputs and rare paths included, and a failure pulled from one is a real failure by definition, not a guess at what might go wrong.

Each has a weakness the other covers. A hand-built set stays fixed while the agent, its prompts, and its users keep changing, so it slowly describes a distribution of traffic that no longer exists. A trace captures one exact interaction, and the class of inputs behind it has to be inferred to make the case generalize. A mature dataset draws on both: a curated core for the behaviors that must always hold, refreshed on an ongoing basis with cases promoted from what production actually sends.

keep reading

More on this.

Can I start running agent evals before I have a labeled dataset?Yes. An LLM judge can score production traces against a rubric with no labeled set required; the labeled cases build themselves from what gets flagged and reviewed.answer →How do I tell whether an eval case has gone stale?Check its capture date against what changed since: a case written before a prompt, tool, or model swap is testing a version of the agent that no longer exists.answer →How many eval cases do I need before I can start?Around 25, pulled from your docs or from production, is enough to begin, per Confident AI's guide to LLM evals for startups. Coverage of distinct behaviors matters more than count.answer →Everything on eval datasetsThe definition, and every question answered under eval datasets.concept →

Send us the traces you already emit.

book a demo