Did the agent finish the job?
Agent task completion grading checks whether your AI agent's output matches the end state a user actually wanted, not just whether each step ran without error. And when the completion rate drops, Tessary tells you which change caused it.
The agent ran clean and still failed
ReliabilityBench, a 2026 benchmark from independent researcher Aayush Gupta, found that agents scoring 96.9% pass@1 under clean conditions dropped to 88.1% once the same tasks were paraphrased the way real users actually phrase requests. In travel booking, agents searched for flights and placed a hold, then stalled before submitting the payment information needed to actually confirm the reservation. Every step ran except the one that closed the loop.
ReliabilityBench, 2026 ↗Reliability is the top blocker
Measuring Agents in Production (MAP), the first large-scale study of AI agents in production, covered 86 deployed systems in 2025.
- 38%
of practitioners rank reliability as their top agent development challenge, ahead of governance and compliance combined
Measuring Agents in Production, 2025
- 75%
of production agent teams interviewed (15 of 20) evaluate without benchmark sets, leaning on A/B tests and direct user feedback instead
Measuring Agents in Production, 2025
Did it run, or did it finish?
A step-completion check confirms the agent ran. None of that confirms the request actually got resolved, and the person who owns the agent usually learns the difference from a customer, not a dashboard.
Step completion
What most evals check
- Confirms the agent didn't error out or time out mid-session
- Tool calls returned syntactically valid responses
- A final message was generated and sent
Task completion grading
What actually matters
- Confirms the end state matches what the user actually asked for
- Verifies a multi-step booking, ticket, or order was carried through, not abandoned partway
- Checks the agent's own success claim against what actually changed in the system
How task completion grading works
Correct a drafted definition of done, grade the full trace against it, and when the rate slips, trace the drop to its cause.
Correct a drafted definition of done
Tessary drafts what "done" looks like from the traces your agent already produced: a ticket closed with a resolution attached, a booking confirmed with a payment reference, a return processed with a refund issued. You correct the draft rather than write it from scratch. The definition lives outside the agent's code and sharpens as bugs are fixed and edge cases land.
Grade the real trace, not the last message
Each production session lands as a typed trace: Session, Turn, Trace, Observation. The completion grader reads the full path the agent took, so a task that stalled three steps before the end gets flagged even when the final reply reads cleanly.
Catch the gap before it ships
A diff classifier reads the next pull request and reports which completion graders are most likely to regress, so a change to checkout logic or escalation rules gets flagged before it reaches production traffic. A check like this only sees changes that arrive through a PR, though. A model provider update or a tool that starts returning different data reintroduces the failure with no deploy at all, which is why the same graders keep watching production.
When the rate slips, find the change that caused it
Tessary watches the completion-rate trend, and when it drops, traces the failing grader back through the session to the change that caused it: a prompt edit, a tool update, an upstream agent's shift, or a model update that never touched your repo. The finding routes as a digest, threshold alert, or PR comment to Slack, Linear, PagerDuty, or wherever the team already triages incidents.
What a completion grader catches
A final-reply check reads one message. A completion grader reads what actually happened.
End-state verification
Checks what actually changed, such as ticket status, order status, or a calendar entry, rather than what the agent's last message claims happened.
Partial-completion detection
An agent that finishes 3 of 4 required steps and still returns a message that reads like success gets flagged at the step where it stopped.
False-success claims
When the agent's final response says a task is done but the trace shows the action never executed, Tessary records the mismatch and ties it to the change that introduced it, whether that was a commit or something that never touched your repo.
A paradox emerges: if reliability is the primary development bottleneck, how do agent systems reach production?
Answered, plainly.
Related pages
Agent evals without labeled data
Start completion graders without a golden set: plain-language failure descriptions become classifiers on your production traces.
Multi-turn agent evaluation
One reply can read fine while the whole conversation misses the user's goal. Grade the full thread.
Evals that evolve with production
Definitions of done drafted from real traffic and kept current as the agent changes.
Know when tasks stop completing, and why
Connect a Langfuse or Braintrust trace source, or send OTLP from your own pipeline, and correct a drafted definition of done for one task. Your existing tracing tells you the completion rate moved; Tessary tells you which change moved it. The first grader runs free against your live traffic, no credit card required.