# Finding RAG answers that aren't grounded in their sources

A customer-facing agent speaks for the company. If it invents a policy or makes a claim the company's documents don't support, that answer can create a dispute or expose the company to legal risk. It can also look like a successful agent run: the answer arrives on time, sounds confident, and produces no error.

We've added a [groundedness classifier](/answers/tessary-groundedness/what-is-tessarys-groundedness-classifier) to Tessary to catch these answers. It checks each eligible RAG answer against the documents the agent retrieved and identifies the specific sentences those documents don't support.

## Checking every answer

We need to check every answer to see when this problem starts happening more often. Using an LLM to read every answer and its sources would make that expensive, so we fine-tuned a model for this narrower task.

Our model builds on the published [lettucedetect](https://huggingface.co/KRLabsOrg/lettucedect-large-modernbert-en-v1) checkpoint. We trained it further on RAGTruth and our verified corpora. It reads the retrieved documents and the full answer together, then scores each sentence for whether it is supported. A sentence can be flagged because it contradicts a document or because it makes a claim none of the documents contain.

On the human-labeled [RAGTruth](https://arxiv.org/abs/2401.00396) test set, the model caught 39% of unsupported answers at a 2% response-level false-alarm rate, with 83% precision. At the point with the best F1 score, it reached 63% recall and 70% precision. These are results on RAGTruth, so teams should expect performance to vary with their own documents and answers.

The model, [`tessaryai/groundedness-classifier-v1`](https://huggingface.co/tessaryai/groundedness-classifier-v1), and its weights are public. It runs on your own hardware, so checking an answer does not require sending its documents to a model API or paying a per-call API fee.

## Knowing when the problem is getting worse

A flagged sentence tells you which claim to inspect. By itself, though, it doesn't tell you whether the agent's behavior has changed. Tessary records every answer it scores, including answers with no flags, and learns the usual flagged rate for each call site. It opens a finding when that rate rises.

![A Tessary finding on a support agent: the share of traces with a flagged answer rose from its usual 2.1% to 6.4%, with a refund answer's unsupported sentences marked beside the retrieved refund policy](/images/blog/groundedness-classifier/groundedness-finding.png)

The image shows what that looks like in a finding. The flagged rate has risen from 2.1% to 6.4%. In one of the affected answers, the agent says customers can get a refund for up to 60 days and receive half their payment after that. The retrieved refund policy instead says refunds are available within 30 days, followed by store credit. Tessary marks the unsupported sentences beside the documents the agent used.

The finding gives the team both the change in rate and the answers behind it. From there, Tessary can run triage and investigate the affected answers with their retrieved documents and the connected agent repository. The investigation looks for likely causes, such as [retrieving the wrong or outdated document](/answers/tessary-groundedness/does-groundedness-check-if-context-is-correct), a prompt asking the agent to answer beyond its sources, or the agent combining information from documents incorrectly.

The groundedness classifier is off by default. Once enabled, it checks eligible RAG answers on your hardware, shows the claims it cannot support, and alerts you when those claims become more common.

---

Source: https://tessary.ai/blog/groundedness-classifier
All posts: https://tessary.ai/blog
From Tessary, agent reliability for AI agents in production: https://tessary.ai