Where do the cases in an agent eval dataset come from?

Cases come from two places. Golden cases are hand-built: someone who understands the agent writes an input and the behavior expected on it, curated and reviewable, which is why they double as the ground truth a grader gets calibrated against. Production traces are the other source: real sessions the agent already handled, messy inputs and rare paths included, and a failure pulled from one is a real failure by definition, not a guess at what might go wrong.

Each has a weakness the other covers. A hand-built set stays fixed while the agent, its prompts, and its users keep changing, so it slowly describes a distribution of traffic that no longer exists. A trace captures one exact interaction, and the class of inputs behind it has to be inferred to make the case generalize. A mature dataset draws on both: a curated core for the behaviors that must always hold, refreshed on an ongoing basis with cases promoted from what production actually sends.

keep reading

More on this.

Can a brand-new eval suite already have a coverage gap?Yes. A coverage gap is the mismatch between what an eval set contains and what production sends, and it can exist the day the set is written, before anything goes stale.answer →Can I start running agent evals before I have a labeled dataset?Yes. An LLM judge can score production traces against a rubric with no labeled set required; the labeled cases build themselves from what gets flagged and reviewed.answer →Can synthetic data replace production traces in an eval dataset?Not as a permanent substitute. Synthetic cases get an eval set running fast, but only production traces show how often a failure actually occurs.answer →Everything on eval datasetsThe definition, and every question answered under eval datasets.concept →

Self-host Tessary.

Free and open source. Point it at the traces your agent already emits.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y