# Why do quality layers built on an LLM judge end up sampling instead of grading everything?

A generative judge makes a paid inference call on every trace it grades, and that cost scales linearly with traffic. A tool that grades everything with one has to either sample or accept a bill that grows with usage, and sampling is the cheapest lever available once a judge is the only grading method in play: cut to 10% of traffic and the bill drops by 10x, no other engineering required.

The tradeoff is what sampling always costs. A failure that shows up in 1% of traffic is unlikely to land in a 10% sample, so whatever the sample misses stays invisible until enough of it accumulates to notice. Doing the cheap grading, deterministic checks or a distilled classifier, on all of it and saving the judge for a flagged slice avoids that tradeoff, but only if something upstream of the judge is actually cheap.

---

Source: https://tessary.ai/answers/eval-costs/why-llm-judge-quality-layers-end-up-sampling
More on Eval costs: https://tessary.ai/answers/eval-costs
From Tessary, agent reliability for AI agents in production: https://tessary.ai