When do I still need a golden set?

You still need a golden set the moment you want to trust the grader, not just the agent. A hand-labeled dataset is what a grader’s precision and recall are measured against, and that measurement is the only way to know whether the judge you built agrees with a person on what counts as a failure. Production traces can’t supply that, because a trace has no expected answer attached until someone writes one.

Day to day, grading production traces without a golden set works fine for catching failures as they happen. A golden set earns its keep on a narrower job: calibrating the instrument, checking a new grader before it’s trusted, and re-checking an old one after the agent, the prompt, or the judge model changes underneath it. Skip that and you’re trusting a judge that’s never been graded itself.

keep reading

More on this.

Can I start running agent evals before I have a labeled dataset?Yes. An LLM judge can score production traces against a rubric with no labeled set required; the labeled cases build themselves from what gets flagged and reviewed.answer →How do I tell whether an eval case has gone stale?Check its capture date against what changed since: a case written before a prompt, tool, or model swap is testing a version of the agent that no longer exists.answer →How many eval cases do I need before I can start?Around 25, pulled from your docs or from production, is enough to begin, per Confident AI's guide to LLM evals for startups. Coverage of distinct behaviors matters more than count.answer →Everything on eval datasetsThe definition, and every question answered under eval datasets.concept →

Send us the traces you already emit.

book a demo