Does running several graders on one trace cost more than running one?

Not proportionally, if the graders share a prompt cache. A cache write costs 1.25x the base input rate and a cache read costs 0.1x, so the first grader on a trace pays to write the input to cache and every grader after it pays a tenth of the read cost for that same input. Run N graders on one trace and the input cost comes out to roughly 1.15 + 0.1N times the base rate, against N times the base rate if each grader paid for its own copy of the input.

At two graders that’s already cheaper than uncached; at ten graders it’s about a fifth of the uncached cost. Break-even lands around N ≈ 1.28 graders per trace, so a single-grader check is usually left uncached.

sources

keep reading

More on this.

What drives the cost of evaluating an AI agent?Cost is driven by which grading methods your traffic hits: deterministic checks cost almost nothing per event, a language model judging a trace does, every time.answer →Which graders belong on every PR and which belong on a nightly run?Cheap, deterministic graders belong on every PR; anything that calls a language model belongs on a scheduled run, because that call is priced per commit.answer →Why does a classifier cost less to run than an LLM judge?A narrow classifier does less work per trace than a generative judge. Training happens once, but checking each new trace still costs compute.answer →Everything on eval costsThe definition, and every question answered under eval costs.concept →

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization
traces
10,000 per calendar month
stored trace data
1 GB
retention
30 days
model credit
$10, one-time, for triage and root-cause analysis
credit card
not required
start for free

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y