What should each eval case record about where it came from?

Each case should record where it came from and when, at minimum: the trace or session id it was pulled from, who confirmed the expected behavior, be it a person or a review process, and the date it was captured. None of that changes what the case tests, but all of it determines whether you can still trust the case a year from now.

Without a source trace, there’s no way back to the original context if the expected behavior ever looks wrong. Without a capture date, there’s no way to tell a case that still matches current behavior from one written against a version of the agent that no longer exists. Provenance is what turns a fixed set of cases into one you can maintain instead of one you eventually have to throw out and rebuild from scratch.

keep reading

More on this.

Can I start running agent evals before I have a labeled dataset?Yes. An LLM judge can score production traces against a rubric with no labeled set required; the labeled cases build themselves from what gets flagged and reviewed.answer →How do I tell whether an eval case has gone stale?Check its capture date against what changed since: a case written before a prompt, tool, or model swap is testing a version of the agent that no longer exists.answer →How many eval cases do I need before I can start?Around 25, pulled from your docs or from production, is enough to begin, per Confident AI's guide to LLM evals for startups. Coverage of distinct behaviors matters more than count.answer →Everything on eval datasetsThe definition, and every question answered under eval datasets.concept →

Send us the traces you already emit.

book a demo