# Eval datasets

An eval dataset is the set of cases an agent gets judged against: inputs paired with the behavior
expected on them. Every score a suite produces is relative to this set, so the dataset defines what
passing means. Cases come from two sources with different properties.

Golden sets are hand-built: curated inputs with expected behavior, labeled by people who understand
the agent. They're precise and reviewable, and they do double duty. They test the agent, and they
supply the ground truth for calibrating automated graders, since a grader's precision and recall are
measured on labeled cases. Their weakness is staleness. The agent changes, its prompts change, and
its users change, while a hand-built set stays fixed, so over time the set describes a past
distribution of traffic. Pass rates on a stale set measure agreement with that past distribution.

Production traces are the other source: real sessions, including the messy inputs and rare paths
nobody thought to write down. A failure observed in a production trace is a real failure by
definition, which grounds the case in traffic the agent actually receives. A trace records one exact
interaction, though, while the failure it exposes is usually a class of inputs. A case built from a
trace stays meaningful across wording changes to the extent it captures that class.

Mature datasets draw on both sources: a curated core covering the behaviors that must hold, which
also serves grader calibration, refreshed with cases promoted from production that keep the set
aligned with current traffic. Provenance, a record on each case of where it came from and when it
was last verified, is what makes staleness visible. Each human label added along the way is fresh
ground truth for calibration, so labeled cases accumulate value beyond the single behavior they
check.

## Questions answered under this concept

- [Can I start running agent evals before I have a labeled dataset?](https://tessary.ai/answers/eval-datasets/can-i-start-agent-evals-without-a-labeled-dataset)
- [How do I tell whether an eval case has gone stale?](https://tessary.ai/answers/eval-datasets/how-do-i-tell-an-eval-case-has-gone-stale)
- [How many eval cases do I need before I can start?](https://tessary.ai/answers/eval-datasets/how-many-eval-cases-do-i-need-to-start)
- [What should each eval case record about where it came from?](https://tessary.ai/answers/eval-datasets/what-should-an-eval-case-record-about-its-source)
- [When do I still need a golden set?](https://tessary.ai/answers/eval-datasets/when-do-i-still-need-a-golden-set)
- [Where do the cases in an agent eval dataset come from?](https://tessary.ai/answers/eval-datasets/where-do-eval-dataset-cases-come-from)

---

Source: https://tessary.ai/answers/eval-datasets
All concepts: https://tessary.ai/answers
From Tessary, agent reliability for AI agents in production: https://tessary.ai