What drives the cost of evaluating an AI agent?

Cost is driven by which grading methods your traffic hits, and how much of that traffic reaches them. A deterministic check, a regex, a schema validator, a rule, runs on ordinary CPU and costs close to nothing per event, so grading every trace with one barely moves the bill. A language model judging a trace is different: it’s an inference call, and that cost recurs on every event it touches. Which model you use, how much context you feed it, and what share of traffic you route to it all multiply together.

Volume just scales whichever mix you’ve already chosen. That’s why architecture decides the bill more than any single model’s price tag: a pipeline that pushes most traffic through cheap deterministic checks and reserves the language model for a flagged slice spends a fraction of one that sends every trace straight to a judge.