What should each eval case record about where it came from?

Each case should record where it came from and when, at minimum: the trace or session id it was pulled from, who confirmed the expected behavior, be it a person or a review process, and the date it was captured. None of that changes what the case tests, but all of it determines whether you can still trust the case a year from now.

Without a source trace, there’s no way back to the original context if the expected behavior ever looks wrong. Without a capture date, there’s no way to tell a case that still matches current behavior from one written against a version of the agent that no longer exists. Provenance is what turns a fixed set of cases into one you can maintain instead of one you eventually have to throw out and rebuild from scratch.

keep reading

More on this.

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization
traces
10,000 per calendar month
stored trace data
1 GB
retention
30 days
model credit
$10, one-time, for triage and root-cause analysis
credit card
not required

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y