How many eval cases do I need before I can start?

Around 25 cases, pulled from your docs or from a handful of production traces, is enough to begin, fewer than most people assume, according to Confident AI’s guide to LLM evals for startups. The point of that first set isn’t statistical significance, it’s having something concrete to run a grader against so you can see whether the grader agrees with you.

Count matters less than what the cases cover. Twenty-five cases that each check a different behavior, a different tool, a different failure mode, tell you more than a hundred that all check the same thing worded differently. Start small, run it, and grow the set from what production actually breaks on rather than trying to anticipate every case up front. The set that matters six months in is the one that kept absorbing real failures, not the one that started biggest.

sources

keep reading

More on this.

Can I start running agent evals before I have a labeled dataset?Yes. An LLM judge can score production traces against a rubric with no labeled set required; the labeled cases build themselves from what gets flagged and reviewed.answer →How do I tell whether an eval case has gone stale?Check its capture date against what changed since: a case written before a prompt, tool, or model swap is testing a version of the agent that no longer exists.answer →What should each eval case record about where it came from?At minimum: the trace it was pulled from, who or what confirmed the expected behavior, and the date it was captured. That record is what makes staleness visible later.answer →Everything on eval datasetsThe definition, and every question answered under eval datasets.concept →

Send us the traces you already emit.

book a demo