What does an eval gate report when a PR has no baseline run?

It reports a one-sided delta, not a zero. Tessary’s CI gate compares the base commit’s latest completed eval run against the head commit’s, grader by grader, and when one side has no completed run at all, that missing comparison stays missing instead of getting treated as a pass. A fabricated zero would read as “nothing changed,” a claim about the base commit’s behavior the gate never actually measured.

The forecast half of the report can still say something with no history behind it: a status of “insufficient history” and a risk score of zero is the gate’s honest way of saying it has no learned correlation yet, not a clean bill of health. The right response to a one-sided report is to run the base commit’s own eval suite so the next comparison has something real to diff against, not to read the gap as a pass.

keep reading

More on this.

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization
traces
10,000 per calendar month
stored trace data
1 GB
retention
30 days
model credit
$10, one-time, for triage and root-cause analysis
credit card
not required

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y