Agent Reliability

Grade code that runs, not prose

Standard evals score coherence and tone. Code generation agent evals run the output: execute it, pass it through tests, and scan for security patterns. Tessary grades what the code does, not what it says, and when a grader trend drops in production, tells you which change caused it.

Find what broke your agentSee the evals SDK
The research

Nearly half of AI-generated code has security flaws

Veracode tested more than 100 language models across 80 coding tasks. The code usually ran; the vulnerabilities came along anyway, and larger, newer models fared no better.

45%

of test cases produced AI-generated code with security vulnerabilities, across 80 coding tasks

Veracode, 2025 GenAI Code Security Report

100+

language models tested, with no meaningful improvement in security from newer or larger models

Veracode, 2025 GenAI Code Security Report
The gap

Text graders measure prose, not code

Code agent output is executable. Standard evals score coherence, accuracy, and helpfulness: none of those dimensions tell you whether the output runs, passes tests, or introduces a vulnerability. When teams apply only text-quality graders to code agent traffic, they measure prose on output that runs in production infrastructure. The correctness signals come from running the code, not scoring the text.

Execution

gives ground-truth correctness that text evals cannot provide

Security

failures are invisible to graders that treat code as prose
Two approaches

Text eval vs code generation eval

Text-quality graders are correct for conversational agents. They are the wrong unit for agents that generate code: the output runs in production infrastructure and can be executed, tested, and scanned.

Text-quality grader

Grades the response, not what it does

  • Cannot detect whether the code executes or passes tests
  • Passes syntactically valid but logically broken output
  • No signal on OWASP vulnerability patterns in the diff
  • Cannot detect when the agent modifies files outside the task scope
Tessary

Code generation grader

Grades what the code actually does

  • Executes the output and checks whether it passes the test suite
  • Scans the diff for OWASP Top 10 patterns before the merge
  • Checks whether the change is scoped to what was asked
  • Tracks test coverage delta to catch untested added behavior
The grader set

Four graders for code agents

The grader set for a code generation agent maps to what code does: does it run, does it introduce vulnerabilities, is it scoped to the task, and does it add untested behavior?

Functional correctness

Does the output execute without errors? Does it pass the test suite? A run that produces syntactically valid but logically broken code needs a grader that can distinguish those two outcomes, not a text-quality score.

Security pattern detection

The OWASP Top 10 is the practical baseline for production grading. SQL injection, cross-site scripting, log injection, and cryptographic failures appear consistently in AI-generated code. This class of failure does not improve predictably with model size.

Diff quality

Coding agents sometimes fix one issue and quietly modify unrelated files, rename symbols across the codebase, or add untested behavior adjacent to the stated task. A diff quality grader checks that the change is scoped to what was asked.

Test coverage impact

When a code agent proposes a change that lowers test coverage, it has likely added behavior without adding verification. Coverage delta is a grader that runs without human judgment and flags changes that functional correctness alone will not catch.
How it works

Catch regressions, then name the cause

Pre-deploy gates catch what is in the diff. A code agent also degrades when nothing in your diff changed: the provider ships a model update, a dependency bumps, a tool starts returning different data. Tessary runs graders on every PR and watches the grader trend on production traffic, so both classes of cause get caught and named.

  1. Connect your trace source

    Send traces via Langfuse or Braintrust, or use the agent evals SDK for direct instrumentation. No re-instrumentation is required if your agent is already sending traces to either platform.

  2. Configure code graders

    Tessary drafts functional correctness, security pattern, diff quality, and test coverage delta graders from your agent's observed output; you correct them rather than write them from scratch. Each grader targets a distinct failure mode that text-quality evals cannot catch.

  3. Gate deploys on grader results

    Tessary reads the incoming diff, scores which graders the change is most likely to trip, and posts a pre-deploy trouble report on the PR. A GitHub Action blocks the merge until grader results are clean.

  4. Trace failures to the cause

    Commit-SHA lineage covers the changes you shipped: every grader result ties back to the PR behind it. When the trend drops and the diff is clean, Tessary traces the failing dimension back through the agent's chain to a cause outside the repo: a tool that started returning different data, an upstream agent whose output shifted, or a model update underneath you.
Attribution

A trend drop with no deploy

Your test-coverage trend drops from a 3% failure rate to 11% on a Tuesday. No deploy that day, and the last diff is clean. Because Tessary reads the trend across grader verdicts rather than any single score, the jump registers as a real change even though the coverage grader is noisy on any one run. The trace goes back through the agent's chain to a package-registry tool that changed its response format the night before. No diff review would have surfaced it, because there was no diff.

0

lines changed in your repo when the trend broke

1

upstream tool whose response format changed, found by walking the chain back
FAQ

Questions about code generation evals

Standard LLM evals score text quality, tone, or factual accuracy. Code generation evals can run the output: execute it, pass it through tests, and scan it for security patterns. That execution path gives you ground-truth correctness checks that text-only evals cannot provide. The grader set is specific to what code does, not what it says.
At minimum: a functional correctness grader (does the code execute and pass the test suite?), a security signal grader (does it match OWASP Top 10 vulnerability patterns?), and a diff quality grader (is the change scoped to what was asked?). Test coverage impact is a useful fourth grader that runs without human judgment.
Wire your graders to CI and run them on every PR diff. Tessary reads the development diff, maps the touched surface to the failure families most likely to fire (tool-call errors, scope creep in the diff, coverage drops), and posts a pre-deploy trouble report before the merge. A GitHub Action can gate the merge if a grader fires. One boundary to know: a CI gate only fires on changes that ship through a PR. A model provider update or a tool returning different data reintroduces the failure with no deploy at all, which is what production trend monitoring is for.
A diff quality grader checks whether an agent's change is scoped to its stated task. Coding agents sometimes fix one issue and silently modify unrelated files, rename symbols across the codebase, or add untested behavior adjacent to the task. A diff quality grader flags changes whose scope exceeds what the task description specified.
Veracode's 2025 GenAI Code Security Report found that AI-generated code introduced security vulnerabilities in 45% of test cases, across 80 coding tasks and more than 100 language models. The code usually ran; the vulnerabilities came along anyway. Larger and newer models performed no better on security, suggesting a structural pattern in how models generate code, not a gap that improves automatically with the next release.
Yes. Tessary connects to Langfuse and Braintrust as upstream sources and pulls a chosen slice of traffic on demand. Once that traffic is ingested, every downstream capability applies to it: graders, alerts, and the pre-deploy CI gate. No re-instrumentation is required. The difference in what you get: Langfuse and Braintrust tell you a grader score moved. Tessary tells you which change moved it, whether that was a prompt edit, a dependency bump, or a model provider update that never touched your repo.
Keep exploring

Related pages
Get started

Grade code agents on what they produce.

Connect your Langfuse or Braintrust source, run the four code graders against live traffic, and when a trend drops, see which change caused it. The first run needs no credit card.

Find what broke your agentAlready on Langfuse or Braintrust? Pull a trace slice without re-instrumenting. When a cause hunt hits a gap in the traces, that gap is the next node to instrument.