Can I test a self-hosted decision model against Tessary's frustration classifier before switching?

Yes. Tessary can run a second, self-hosted decision model in shadow beside whichever provider is already scoring frustration, sending it the same turns and storing both verdicts side by side, with nothing a user sees changing. Every turn the live provider scores gets re-sent to the shadow exactly as the provider saw it, and the comparison is a straight count: how often the shadow’s flag agrees with the provider’s on the same turns.

The shadow doesn’t need to be up all the time either. A connection failure or server error on its end just leaves that turn’s comparison for the next pass, and an outright refusal from the shadow is recorded as a refusal rather than resent forever, so a server that sleeps most of the day doesn’t pile up retries.

That’s the harness for answering whether a self-hosted System One model is ready to take over from a hosted provider, on your own traffic, before anything about production actually changes.

sources

keep reading

More on this.

Can the frustration classifier tell why a user is upset, or just that they are?The classifier only says that a user is unhappy with the assistant. Root cause analysis on the case says why, grouping the frustrated conversations by what the agent did.answer →Does a calm bug report count as frustration?Usually not. A matter-of-fact bug report reads as neutral. What counts is a user unhappy with the assistant, in their own words or in re-asking after a clear failure.answer →Does Tessary's frustration classifier see the whole conversation, or just the last few turns?It reads the two turns right before the one it's scoring, not the whole conversation, so frustration that hasn't recurred in a few turns won't factor in.answer →Everything on tessary frustrationThe definition, and every question answered under tessary frustration.concept →

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization
traces
10,000 per calendar month
stored trace data
1 GB
retention
30 days
model credit
$10, one-time, for triage and root-cause analysis
credit card
not required
start for free

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y