# How do I catch an agent that claims a task is done when it isn't?

Grade the state a session actually left behind, not the agent's closing message. A completion grader compares an end state, a payment reference, a ticket status, a calendar entry, against what the user asked for, so the words "done" or "confirmed" in a transcript carry no weight by themselves.

[ReliabilityBench](https://arxiv.org/abs/2601.06112) shows why the distinction matters. Its travel domain, built on tools like `search_flights`, `hold_flight`, and `confirm_booking`, scores 87.5% pass@1, and the paper's own failure analysis names the gap directly: agents fail to provide payment information for booking confirmation, a missing-context problem, not a broken call. Picture what that looks like in a trace: every step up to `confirm_booking` can return a valid response, so a checker that only reads step outcomes passes the run, even though `confirm_booking` never fires and no reservation exists.

The fix is to check the system the agent was supposed to change, not the trace of calls that got it there. If the booking system shows no reservation, the run failed, whatever the last message claimed.

---

Sources:
- ReliabilityBench (arXiv:2601.06112): https://arxiv.org/abs/2601.06112 (fetched 2026-08-25)

Source: https://tessary.ai/answers/failure-modes/how-do-you-catch-an-agent-claiming-a-task-is-done
More on Failure modes: https://tessary.ai/answers/failure-modes
From Tessary, agent reliability for AI agents in production: https://tessary.ai