Eval datasets

An eval dataset is the set of cases an agent gets judged against: inputs paired with the behavior expected on them. Every score a suite produces is relative to this set, so the dataset defines what passing means. Cases come from two sources with different properties.

Golden sets are hand-built: curated inputs with expected behavior, labeled by people who understand the agent. They're precise and reviewable, and they do double duty. They test the agent, and they supply the ground truth for calibrating automated graders, since a grader's precision and recall are measured on labeled cases. Their weakness is staleness. The agent changes, its prompts change, and its users change, while a hand-built set stays fixed, so over time the set describes a past distribution of traffic. Pass rates on a stale set measure agreement with that past distribution.

Production traces are the other source: real sessions, including the messy inputs and rare paths nobody thought to write down. A failure observed in a production trace is a real failure by definition, which grounds the case in traffic the agent actually receives. A trace records one exact interaction, though, while the failure it exposes is usually a class of inputs. A case built from a trace stays meaningful across wording changes to the extent it captures that class.

Mature datasets draw on both sources: a curated core covering the behaviors that must hold, which also serves grader calibration, refreshed with cases promoted from production that keep the set aligned with current traffic. Provenance, a record on each case of where it came from and when it was last verified, is what makes staleness visible. Each human label added along the way is fresh ground truth for calibration, so labeled cases accumulate value beyond the single behavior they check.