How do you score turn-taking smoothness when a user interrupts the agent?

With a dedicated grader on the turn itself, because a transcript-only eval can score every one of a caller’s replies as accurate and still miss that the caller got talked over. IHBench, a 2026 benchmark on post-interruption recovery in voice agents, found GPT-family models correctly resume an utterance after a backchannel like “mm-hm” only 7% to 31% of the time, against 62% to 68% for Gemini 2.5. None of that shows up in the words: the resumed sentence still reads as a complete, plausible reply.

Grading it needs the trace to carry turn structure, who was speaking when, and where a backchannel or a real interruption landed inside an utterance. Instrumentation that records only the final transcript has nothing for a turn-taking grader to read, which is a gap in what got captured, not something a better grader can work around.

Two things get graded separately: how fast the agent yields the floor, and whether it resumes at the right point afterward. An agent can do one well and still fail the other.

sources

keep reading

More on this.

How do you measure whether a grader itself is accurate?Run it against a human-labeled set with a known right answer and compare its verdicts to those labels; the resulting precision and recall are its accuracy.answer →Is an agent that correctly refuses a task a completion failure?To a task-completion grader, yes: a required refusal and an abandoned task both show no state change, and the grader can't tell them apart.answer →What does an eval score actually claim about my agent?It's a claim about the eval set the graders ran on, not the agent as a whole; it says nothing about a failure mode no grader in the set checks for.answer →Everything on gradersThe definition, and every question answered under graders.concept →

Two ways to run Tessary.

Tessary is an open-source agent reliability platform. Cloud and self-hosted run the same workflow on the OpenTelemetry traces your agent already emits.

Tessary Cloud

We host it for you. Send your first trace with nothing to deploy and no model key.

what's includedper organization
traces
10,000 per calendar month
stored trace data
1 GB
retention
30 days
model credit
$10, one-time, for triage and root-cause analysis
credit card
not required
start for free

Self-hosted Tessary

Run the open-source code on your own infrastructure with one command. Add your own model key for triage and root-cause analysis.

Self-host Tessary for me by following https://github.com/tessaryai/tessary/blob/main/setup.md

docker compose -f oci://docker.io/tessaryai/tessary:compose up -d -y