Does passing your eval suite before launch mean an agent will stay reliable?

No. An eval suite is a snapshot: it grades the agent against the model, prompts, and tools as they existed the day you ran it. None of those stay fixed. Teams keep shipping changes to prompts and tools, and the underlying model can change on the provider’s side with no deploy on yours, so the agent you tested is not the agent running next month. A suite that passed once says nothing about what happens after the next silent model update or the next tool that changes its response format. Treating a passing eval run as a one-time certification misses this: reliability is a property of the agent over time, not a property of one version at one moment. The only way to know an agent is still reliable is to keep watching what it does in production, not to trust the score it earned before launch.

keep reading

More on this.

How do you measure agent reliability in production?As a rate across many runs, such as the share of production sessions that behave as intended, tracked over a rolling window, rather than a pass or fail judged on a single run.answer →Is agent reliability the same as uptime?No. Uptime measures whether the agent's process responded without erroring; agent reliability measures whether what it did was correct, and a wrong answer usually returns a healthy status.answer →What causes an AI agent to become less reliable?Any change to the model, prompts, tools, or data an agent depends on can degrade it, and the model itself can change on the provider's schedule with no deploy on your side.answer →Everything on agent reliabilityThe definition, and every question answered under agent reliability.concept →

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization
traces
10,000 per calendar month
stored trace data
1 GB
retention
30 days
model credit
$10, one-time, for triage and root-cause analysis
credit card
not required
start for free

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y