Turn Production Failures Into Regression Tests: The Step Engineers Skip

According to LangChain’s evaluation resource, the bottleneck in LLM evals is not which scoring technique to use, but rather building the operational workflows that turn production failures into reproducible test cases. The workflow most teams are missing is the production trace regression test: extract the failing span, write a grader for the behavior it exposed, and wire the grader into CI so the failure class has a gate.

The issue is not that engineers fail to fix the bug. It is that fixing the bug and adding regression coverage for the bug are two different steps, and most teams only do the first.

Why the Same Bug Comes Back

Without a grader in CI, a prompt change or model update reintroduces the failure class with no gate to catch it. A prompt change ships on Wednesday and introduces a context-assembly error. Four sessions produce a nonexistent tool call before anyone files a ticket. The engineer finds the trace on Thursday, patches the prompt, closes the incident, and moves on.

Three months later, a similar change reintroduces the failure in a slightly different form. Because no grader ever encoded the failure class, CI cannot tell the difference between the fixed state and the broken state.

The missing step is between “found the failing trace” and “added it to the test suite.” That step is not optional for production coverage; it is the part that makes the fix stick.

Extracting the Right Span From the Failing Trace

Not every span in a failing trace is what you want to capture. A session with a tool hallucination might span five turns; the grader needs the single turn where the hallucination occurred and the specific tool call that went wrong. This is the same motion as replaying an agent failure: isolate the exact turn, with the exact inputs that produced it.

Export the session from your trace source (Langfuse and Braintrust both export structured spans). Identify the turn containing the failure: look for the model call whose output produced the incorrect behavior, not just the final response. Pull the inputs to that call, including what the context assembly step delivered. That input is the fixture for your grader.

If you are already connected to Tessary, the failing turn is already captured with its session and typed spans. Pull it directly, without a separate export step.

Writing the Grader

The common mistake is writing a grader that checks the specific bad output rather than the failure class the trace exposed. A grader that flags a particular wrong response string tests for that exact output. It misses every future variant that exposes the same behavioral gap through a different surface.

Three failure modes map to specific grader checks:

• Tool hallucination: check whether the called tool name exists in the available set, not whether the model produced the exact wrong string. A future prompt change may produce a different nonexistent tool name; set-membership catches it, string-matching does not.
• Missing field in retrieval: check field presence. If the trace shows a required field missing, the grader asserts it is present in each response.
• Prompt constraint violation: check constraint adherence. If the trace shows the agent ignoring a constraint, the grader verifies the constraint was followed.

You can also write the grader as a sentence. Describe the failure in plain language, “the agent calls a tool that is not in the registered tool set”, and Tessary compiles that description into a classifier that runs against every span in the eval run. The grader stays readable and adjustable as the agent evolves.

Each grader written this way also encodes a piece of what the agent is supposed to do, stored outside the agent’s code. Ten incidents in, the suite holds a working definition of correct behavior that nobody had to author from scratch. Tessary drafts graders from the agent’s observed behavior and you correct the drafts, which costs far less than writing evals by hand, and the recorded intent sharpens with every bug fixed.

Wiring It Into CI

The grader becomes regression coverage when it runs on every deploy, not just once against the incident trace.

Add it to the eval suite that runs in your CI gate. Scope the gate to the call sites the grader covers: a grader for a retrieval failure does not need to trigger on PRs that only touch the summarization layer. The pattern that scales is a subset of graders per affected call site on the PR gate, with the full suite on a nightly batch. A grader written from a production trace fits the PR gate because its pass/fail verdict is unambiguous.

The GitHub Action step is short:

# .github/workflows/evals.yml
- name: Run regression graders
  uses: tessary/evals-action@v1
  with:
    suite: regression
    scope: changed-call-sites
    fail_below: 0.95

If the grader fails, the gate posts the result on the PR and the merge is blocked. Each verdict is attached to the commit SHA that triggered it, so when a grader fails after a future deploy, the investigation starts at the right code change rather than working backward from user complaints.

Where the CI Gate Stops

A CI gate only fires on changes that ship through a PR. A model provider update, an upstream agent whose output shifts, or a tool that starts returning different data reintroduces the same failure class with no deploy and nothing in any diff. No fix is permanent on its own.

So the same grader keeps running against production traces after the merge. Detection there reads the trend across verdicts over time, not any single score: an imperfect grader misfires at a stable rate, so a failure rate that jumps from 3% to 11% signals a real change. When the trend moves, Tessary traces the failing quality dimension back through the agent’s chain to the change that caused it, whether that is a prompt edit from last week’s PR or a provider update that never touched your repo. The commit SHA on each CI verdict is one input to that lineage, not the whole of it.

Langfuse and Braintrust hold the traces and tell you the score moved; Tessary tells you which change moved it. Connect a source, pull your first failing trace, and write the grader from it, free and with no credit card required.

Find what broke your agent