From Tessary
Notes on keeping agents reliable.
Signals over real production traffic, alerting to the tools you already run, and reading the diff before a deploy. Monitoring and evals are one graph here, so this is the same place reliability gets judged.
LLM Judge Drift: How to Tell What Actually Broke
LLM judge drift is when the model behind a hosted grader changes behind its API, making a moving eval trend ambiguous between a worse agent and a worse judge.
What an AI Agent Production Incident Actually Looks Like
An AI agent production incident rarely trips a normal alert: requests still succeed while the output quietly worsens, and the cause can sit outside your repo.
LLM Judge Calibration: How to Check If a Grader Is Right
LLM judge calibration is the process of checking a grader's verdicts against human labels to see whether it agrees for the right reasons, not just often.
How to Write LLM Agent Graders
How to write LLM agent graders that hold up in production: build the rubric, pick LLM-as-judge or a deterministic check, then calibrate against labeled traces.
MCP Tool Call Failures Are Silent by Default
An MCP tool call can fail without an error: the response validates, the agent moves on, and the real action never happened. How to catch it, and trace the drop to the server that changed.
How to Close the Agent Regression Test Coverage Gap
Your eval suite can pass while the exact bug you just fixed comes back. Why regression graders belong in the same commit as the patch, with Anthropic's own postmortem as the case study.
Model Version Regression Evals: A Pre-Upgrade Checklist
Model version regression evals compare your agent's graders across the outgoing and incoming model, call site by call site, before you upgrade. Plus what to do when the provider upgrades for you.
Agent Cost Monitoring: What a Cost Spike Usually Means
A cost spike is often the first sign an agent broke: retries, context bloat, and runaway loops show up in spend before users complain. Trace the spike to the change that caused it.
Eval Dataset Drift: Why Golden Test Sets Stop Catching Failures
Your eval suite passes while production degrades. Three forms of dataset drift create that false confidence. How to detect each one from your production traces.
Eval Coverage Gap: Why Passing Scores Don't Mean Passing Users
Why a 95% eval pass rate coexists with live failures: graders test the cases you imagined, not the distribution users send. How to find the gap from your traces.
Why Multi-Agent Orchestration Failures Go Undetected
Multi-agent systems fail on 41% to 86.7% of benchmark tasks across seven frameworks. This maps the four failure classes span monitoring misses, and how to trace each one to its cause.
Agent Hallucination Monitoring: How to Catch Plausible Wrong Output Before Users Do
How to catch plausible wrong agent output before users do, and trace a firing grader back to the change that caused it. Error rates alone miss this failure class.
Turn Production Failures Into Regression Tests: The Step Engineers Skip
The production trace regression test workflow: extract the failing span, write a grader for the failure class, wire it into CI, and keep it running where CI cannot see.
How to Detect Silent LLM Agent Failures in Production
Silent failures in LLM agents return HTTP 200 and never trigger an alert. Four failure classes, the graders that catch each, and how to trace a firing signal to its cause.
Agent Evals CI Pipeline: What to Run on Every PR
A practical guide to the agent evals CI pipeline: two-tier PR vs. nightly strategy, pass/fail thresholds, run time under 8 minutes, and the regressions no diff-triggered gate can catch.
Production Traces vs Golden Dataset LLM Evals
Golden-dataset LLM evals test the product as it was. Production traces test it as it runs today, and carry the evidence to show which change made a grader fail.
Agent Regressions from Non-Prompt Code Changes
Agent regressions from non-prompt code changes are the class prompt-only evals miss. Here is how graders detect them, and how the failing dimension gets traced back to the change that caused it.
Why Agent Evals Go Stale and Miss the Regressions That Matter
Agent evals maintenance regression: why static suites go stale as the agent changes, and how a flagged drop gets traced to the change that caused it.