# Where do the cases in an agent eval dataset come from?

Cases come from two places. Golden cases are hand-built: someone who understands the agent writes an input and the behavior expected on it, curated and reviewable, which is why they double as the ground truth a grader gets calibrated against. Production traces are the other source: real sessions the agent already handled, messy inputs and rare paths included, and a failure pulled from one is a real failure by definition, not a guess at what might go wrong.

Each has a weakness the other covers. A hand-built set stays fixed while the agent, its prompts, and its users keep changing, so it slowly describes a distribution of traffic that no longer exists. A trace captures one exact interaction, and the class of inputs behind it has to be inferred to make the case generalize. A mature dataset draws on both: a curated core for the behaviors that must always hold, refreshed on an ongoing basis with cases promoted from what production actually sends.

---

Source: https://tessary.ai/answers/eval-datasets/where-do-eval-dataset-cases-come-from
More on Eval datasets: https://tessary.ai/answers/eval-datasets
From Tessary, agent reliability for AI agents in production: https://tessary.ai