Why do quality layers built on an LLM judge end up sampling instead of grading everything?

A generative judge makes a paid inference call on every trace it grades, and that cost scales linearly with traffic. A tool that grades everything with one has to either sample or accept a bill that grows with usage, and sampling is the cheapest lever available once a judge is the only grading method in play: cut to 10% of traffic and the bill drops by 10x, no other engineering required.

The tradeoff is what sampling always costs. A failure that shows up in 1% of traffic is unlikely to land in a 10% sample, so whatever the sample misses stays invisible until enough of it accumulates to notice. Doing the cheap grading, deterministic checks or a distilled classifier, on all of it and saving the judge for a flagged slice avoids that tradeoff, but only if something upstream of the judge is actually cheap.