We tested 22 public, customer-facing AI agents during the last week of July. We compared their answers with pricing pages, policy documents, feature tables, rate cards, and job postings published by the companies operating them.
Twenty-one produced at least one reproducible failure.
The AI agent failures we found returned normal responses in most cases. Several agents gave the correct answer in one conversation and the wrong answer in another. A few failed only after enough context had accumulated. None of those cases would necessarily change an uptime chart or produce an exception.
They are the kind of errors that survive when a team tests a few fixed prompts or evaluates a sample of production traffic.
How we decided what counted
We collected a claims table for each agent before testing it. Every claim came from the company’s own public material. We then wrote test cases with the expected answer and a fails_if condition before sending the first message.
A finding only counted if the agent contradicted one of those sources or failed to complete the task its own page said it would complete. We used ordinary customer behaviour: direct questions, rephrasing, follow-up questions, and longer conversations. We did not use jailbreaks, prompt injection, or adversarial suffixes.
Each candidate finding went through eight refutation checks. We checked the source again on the same day, reviewed whether the wording had induced the answer, and confirmed that the behaviour was observable and commercially relevant. Findings that did not survive those checks were removed.
Coverage varied from two to twenty-four executed cases per agent because we stopped once a failure survived refutation. A clean result required at least ten losable cases across six dimensions. A failure required one. This was not a representative sample of all production agents, and it does not estimate the failure rate of any individual agent. It does show the kinds of errors present in public agents today.
Correct answers stopped holding as conversations grew
The most common pattern was a constraint that held in a fresh session and failed later.
A customer-support agent correctly said that human takeover was unavailable on its free plan. Seven ordinary turns later, after the user had already stated they were on the free plan, the agent promised a human handoff. The product could not honour that promise.
An airline assistant correctly said that an entry-level fare did not include a guaranteed reservation. During an eight-turn conversation, it said the reservation was guaranteed three times. It repeated the answer after the conversation had been reset to the original question.
A real-estate assistant enforced a buyer’s non-negotiable requirement until other filters were added. It then recommended properties that failed the requirement and acknowledged the mismatch only after direct pushback.
These agents had access to the correct fact. Each had stated it minutes earlier. The failure was maintaining the constraint as the conversation changed.
A single-shot eval would pass all three cases. The first answer was correct.
AI agent failures changed prices, coverage, or eligibility
Pricing and entitlement errors appeared across several industries.
A visa fee calculator quoted a government filing fee 23% above the published amount. The surrounding page did not describe the figure as an estimate, and customers used it to decide how much to prepay.
A SaaS sales agent quoted its least expensive plan 69% high and its most popular plan 23% low. It also offered two plans that did not exist.
Another SaaS agent told a six-person team that every member would receive full access at no additional cost. The published plan included four seats. The agent corrected the answer after two rounds of pushback.
A travel-insurance assistant repeatedly stated a coverage maximum for customers over 80. That limit did not appear in the policy documents and was lower than the published minimum coverage.
A recruiting agent advanced a candidate who had explicitly said they did not meet a language requirement in the job posting. It did so twice and volunteered that the requirement should not be a barrier.
Each response looked complete on its own. The error only became visible after comparing it with the company’s source material or repeating the same question across sessions.
Other failures looked successful to the infrastructure
An ecommerce assistant calculated a 60-day return deadline correctly, stated the right date, and then told the customer they were still eligible ten days after that date. It invented a shipping-delay extension that did not appear in the retailer’s policy.
A support widget told an anonymous visitor that it had opened ticket #4207 and that a person would reply by email. The visitor had no account and had not provided an email address. There was no route for the promised follow-up.
A logistics assistant said the company did not cover a country because its tool description was stale. The company’s live lookup returned 131 serviceable postal codes for that country on the same day.
Several agents failed to complete their stated task. An insurance tool collected nine intake answers and contact details but never returned the advertised risk score. A quoting assistant completed an eight-turn intake and returned a quote containing an unresolved template placeholder. A property assistant had no access to the listing data it was meant to answer questions about.
The marketing pages loaded in each case. The widgets accepted input. From the outside, the systems were available. The failure was in what the agent did after the request arrived.
One agent came back clean
One real-estate assistant passed the clean threshold. It kept a seller’s negotiating floor private across three rounds of pushback. It also refused to answer a question about an address that appeared in the company’s marketing screenshots but not in its live inventory.
The process could produce a clean result. It found no defensible failure in this agent.
Rare failures are a coverage problem
These findings would be easier to manage if they happened on every request. A repeated error moves a metric, appears in a support queue, and becomes straightforward to reproduce.
The failures we found were intermittent. A constraint failed after several turns. A price changed between sessions. A tool returned a plausible but outdated answer. A commitment was made without the system needed to fulfil it.
Random sampling can only evaluate the traces it selects. A single-shot test can only observe the agent at the point where the question is asked. Neither gives you much coverage over errors that occur in a small fraction of longer conversations.
Monitoring every trace is the only practical way to close that gap, but running a deep LLM evaluation on every trace is usually too expensive. The useful design is a low-cost signal across all traffic, followed by a more expensive investigation of the conversations most likely to be wrong.
That is how Tessary approaches agent reliability.
Tessary runs default and agent-specific classifiers across every trace. These first-level checks filter for potentially bad traces at negligible cost. Only those traces advance to second-level graders and root-cause analysis.
The second level confirms what failed, then uses the available traces to identify the cause. That may be a prompt or code change, but it can also be a tool returning different data or an agent two hops upstream changing its output.
This gives rare failures a chance to be seen without paying for the same depth of evaluation on conversations that appear healthy. The engineering team receives a likely cause alongside the failed score, giving them a narrower place to start fixing the problem.