Eval costs

Eval costs are the arithmetic of judging an agent's traffic. Grading methods differ in per-event cost by orders of magnitude, and at scale those differences dominate the bill. A deterministic check costs effectively nothing per event. Grading with a language model pays for inference on every event, and that price varies widely with the model chosen.

The gap comes from where costs land. Some methods carry their cost per event: every trace a model judges incurs an inference charge that recurs with traffic. Others carry their cost up front: a trained classifier's cost is paid once at training time, so grading an additional trace with it costs close to nothing. Because the per-event methods carry nearly all the marginal cost, total spend tracks the share of traffic that reaches them multiplied by traffic volume.

This arithmetic decides what evaluation is feasible. Low-probability failures surface only when a large share of traffic gets examined, so the per-event cost of grading caps how much traffic can be examined, and with it which failures can be observed at all.