Grade RAG retrieval and generation separately
RAG pipelines fail in two independent places: retrieval brings back the wrong or incomplete chunks, or generation misuses chunks that were actually correct. Grading both separately, from production traces, shows which one broke on a given turn, without a hand-labeled golden dataset. And when the retrieval score starts sliding, Tessary traces the drop to the change that caused it: a re-index, an embedding swap, a chunker update.
One passing score, one hidden retrieval miss
In an April 2026 Towards Data Science account, engineer Priyansh Bhardwaj described a Slack message from his compliance team three weeks after his team shipped an internal knowledge base. The assistant had answered a contractor question using only half of a two-part policy, because a chunk boundary fell between the general rule and its exception, and only the first chunk came back on that query. Generation had used the context it was given faithfully. Retrieval brought back the wrong half, so a single end-to-end score would have shown a passing turn: the model never hallucinated anything, it just never saw the exception clause.
Your Chunks Failed Your RAG in Production, Towards Data Science ↗Three ways a RAG turn can fail
A pipeline's health splits into three separate measures, and a pipeline can pass one and fail another on the exact same turn.
Context precision
Whether the chunks a retriever returns are actually relevant to the question. Low precision means the retriever is surfacing noise alongside the useful chunks.
Context recall
Whether the retriever found all the relevant information available, not just some of it. High precision with low recall means the retriever is accurate but incomplete.
Faithfulness
Whether the generated answer stays true to whatever context it actually received. A model can score well here while retrieval quietly handed it the wrong or partial chunks.
Low answer quality alongside high retrieval quality points at the prompt or the model. High answer quality alongside low retrieval quality points at chunking or the embedding index. A single combined score collapses that distinction.
From a retrieval call to the change that broke it
Braintrust's guide above defines the metrics well. The step it leaves to you, getting from a moved score to the change that moved it, is the step Tessary automates. It starts with capturing the retrieval call and the generation call that follows it in the same trace.
Land the trace
Push production traffic over OTLP, or add the TypeScript or Python SDK, so each retrieval call and the generation call that follows it are captured as typed observations in the same trace. Already on Langfuse or Braintrust? Connect it as an upstream source and pull the slice that contains your retrieval calls.
Write the grader in plain language
Describe the specific retrieval or generation failure you’re watching for, such as "no chunks above the relevance threshold" or "answer uses information not present in any retrieved chunk," or start from the graders Tessary drafts from your observed traffic and correct them. No labeled dataset required to start.
Watch the two trends separately
Route an alert to Slack, PagerDuty, or Linear when a grader fires, and track whether retrieval or generation is the one drifting as your knowledge base or your prompts change. A single verdict can be wrong; Tessary reads the trend across turns, so even an imperfect grader reliably flags a real recall drop.
Trace the slide to the change that caused it
When the retrieval trend starts sliding, Tessary walks it back to the change behind it, Tuesday’s re-index, the embedding model swap, the new chunker config, and names that change in the alert. Most of these causes never appear in a code diff, which is why the trace-back starts from the trend rather than the repo.
One score vs two verdicts
A single end-to-end score collapses retrieval and generation into one number. Grading them separately on the same trace shows which one actually broke.
One combined score
Hides which stage broke
- A retrieval miss and a faithful generation collapse into one passing or failing number
- Chunking failures pass silently because faithfulness looks fine
- Stale index drift doesn’t show up as an obviously wrong answer
Retrieval and generation scored separately
Same trace, two verdicts
- High retrieval, low generation points at the prompt or the model
- Low retrieval, high generation points at chunking or the embedding index
- Each verdict lands on the same trace as the call it grades
Three failure modes retrieval grading catches
Each of these passes a combined generation score every time. Grading retrieval on its own is what surfaces them.
- Silent chunking failures
- A boundary that splits a rule from its exception, or a table from its footnote, passes generation grading every time, because the model is faithful to whatever partial context it received.
- Stale index drift
- Recall on a query type can degrade after a re-index or an embedding model swap without any single generation turn looking obviously wrong. Nothing in your code diff explains it, because the cause lives in the index, not the repo. Tessary attributes the recall drop to the re-index event itself, so the fix is not a code archaeology session.
- Context ignored, not missing
- Retrieval can score well while generation quietly drops the relevant chunk from its answer, which a retrieval-only metric would never catch.
Questions about RAG retrieval grading
Related pages
LLM agent behavior drift
How quality slides with no deploy, and how a grader trend catches it before users do.
Agent evals without labeled data
Start grading from the traces you already have, before any golden dataset exists.
Evals that evolve
Graders drafted from observed behavior that sharpen as your pipeline and traffic change.
Start grading your RAG traces.
Point Tessary at the traces your retriever and generator already produce. The first split score shows which stage broke; the trend behind it shows which change broke it.