Can I start running agent evals before I have a labeled dataset?

Yes. You don’t need a hand-labeled dataset to start grading an agent: write a rubric for what a good and a bad answer look like, and an LLM judge can score real production traces against it today. Nothing has to exist before you can start.

The labeled set builds itself as a side effect of doing that. Traces the judge flags, or that a human overrides, go into review, and once someone confirms what the agent should have done, that trace becomes a labeled case grounded in a real failure rather than one invented at a desk.

What this doesn’t give you is confidence in the judge itself. A rubric-based grader can be wrong in ways nobody notices until its verdicts get checked against real labels, which is a separate problem from grading the agent, and the reason a golden set still matters even once you’ve started.

keep reading

More on this.

How do I tell whether an eval case has gone stale?Check its capture date against what changed since: a case written before a prompt, tool, or model swap is testing a version of the agent that no longer exists.answer →How many eval cases do I need before I can start?Around 25, pulled from your docs or from production, is enough to begin, per Confident AI's guide to LLM evals for startups. Coverage of distinct behaviors matters more than count.answer →What should each eval case record about where it came from?At minimum: the trace it was pulled from, who or what confirmed the expected behavior, and the date it was captured. That record is what makes staleness visible later.answer →Everything on eval datasetsThe definition, and every question answered under eval datasets.concept →

Send us the traces you already emit.

book a demo