Tessary evaluates every production trace to find issues, and one of the most useful signals is when a user is unhappy with the agent. Product and engineering teams want to find these conversations, understand what went wrong, and fix it. At scale, they are difficult to find. Complaints and negative ratings only capture a small percentage of them, while manually reviewing every conversation is not practical.
We wanted to build a classifier that could identify this signal across every conversation. This meant solving for more than accuracy. The classifier needed to work across agents with different conversational styles, and it needed to be cheap enough to run continuously. A model that performs well but costs as much as the original agent run does not work when it needs to evaluate every trace.
Detecting whether the user is unhappy with the agent
Detecting negative emotion is not enough. A user discussing a rejected insurance claim may be frustrated with the insurer, the hospital, or the situation itself. That does not mean the agent has done anything wrong. The classifier needs to understand both that the user is unhappy and what that unhappiness is directed at.
For each eligible user turn, we extract the context relevant to the user’s reaction and ask the model to determine whether the user is unhappy with the agent, unhappy about something else, or not unhappy. The model returns a probability for each answer, and Tessary flags the conversation when the probability that the user is unhappy with the agent crosses the project’s threshold. This distinction helps us avoid flagging conversations simply because they contain negative language.
Training our own models
We started by training multiple versions of Ettin-68M and ModernBERT using different datasets. The goal was to build a narrow classifier that could perform this task at a low enough cost to run across every conversation. The models performed reasonably well on conversations similar to their training data, but their performance dropped quickly when we tested them on a different conversational dataset.
We evaluated the models on DialMAGE, which contained 803 conversation turns, including 257 unhappy turns. At a 6% false-alarm rate, Ettin-68M reached 14.9% recall, while ModernBERT reached 24.4%. This was not reliable enough for Tessary because the classifier needs to work across agents built by different teams, with different users, domains, prompts, and conversation structures.
Trying Jev
Over the last week, we tested TypeSafe’s Jev model on the same task. Jev is a decision model that takes some context and a set of possible answers, then returns a probability for each answer instead of generating a written response. This maps well to the narrow decision our classifier needs to make.
We compared Jev with Sonnet 5, which we had used as the teacher for our smaller models. At a 6% false-alarm rate, Jev reached 68.4% recall and 85.6% precision. Sonnet reached 70.6% recall and 84.7% precision. For this task, the two models posted basically the same numbers.
The cost difference was much larger. Jev costs about $0.04 per 1,000 messages, while Sonnet costs $1.34 for the same volume. This makes Sonnet roughly 34 times more expensive. Since this classifier runs across every eligible conversation rather than a sample, that difference determines whether continuous evaluation is practical.
Jev gave us close to Sonnet-level performance at a cost that allows us to evaluate every trace. It also performed much better than the smaller models when the style of conversation changed, removing the need to keep training and maintaining separate models for different distributions.
From a frustrated conversation to a root cause
The classifier now runs across every eligible conversation in Tessary. Finding individual conversations is useful, but the more important signal is when frustration begins increasing. Tessary learns the normal frustration rate for each call site and opens a case when the current rate moves meaningfully above that baseline.
The case contains the affected conversations and the change Tessary detected. Teams can inspect the specific moments where users became unhappy instead of manually searching through transcripts or waiting for complaints. Tessary can then run root-cause analysis with the connected agent codebase to identify what is causing the frustration.
This creates a path from a user expressing unhappiness to the conversations where it happened, the change in behaviour, and the likely cause inside the agent. The team can move from knowing that users are frustrated to understanding what needs to be fixed.
What this changes for agent reliability
This classifier reinforced how we think continuous agent reliability should work. Sampling makes evaluation cheaper, but creates a chance of missing low-frequency issues. Running a large model across every trace provides better coverage, but becomes expensive quickly.
Cheap decision models provide a useful first layer. They can evaluate every trace for narrow signals, while larger models are reserved for the smaller number of findings that need validation, investigation, or root-cause analysis. This lets Tessary watch everything without paying for deep analysis on everything.
We have now shipped Jev as the model powering Tessary’s frustration classifier. It finds conversations where users are unhappy with the agent, detects when frustration increases, and helps identify the cause.