How many eval cases do I need before I can start?

Around 25 cases, pulled from your docs or from a handful of production traces, is enough to begin, fewer than most people assume, according to Confident AI’s guide to LLM evals for startups. The point of that first set isn’t statistical significance, it’s having something concrete to run a grader against so you can see whether the grader agrees with you.

Count matters less than what the cases cover. Twenty-five cases that each check a different behavior, a different tool, a different failure mode, tell you more than a hundred that all check the same thing worded differently. Start small, run it, and grow the set from what production actually breaks on rather than trying to anticipate every case up front. The set that matters six months in is the one that kept absorbing real failures, not the one that started biggest.