Failure replay

Failure replay is the practice of reproducing a production failure with its full original context, so that a proposed fix can be verified against the case that actually happened.

It exists because agent failures are context-dependent. A failure comes from a specific conversation history, a specific set of tool results, and specific state. The same prompt with a paraphrased message or slightly different state will often behave differently, so a fix checked against an approximation of the failure can pass while the original case still fails.

Replaying a failure means capturing enough of the trace to re-run the failing turn faithfully: the message history the model saw, the tool outputs as they were returned at the time, and the configuration that was active. Tool outputs have to be recorded because the live systems behind them change state, and a later call would return different results. With that context preserved, a proposed fix runs against the real failing case, so its verdict reflects the actual failure.

The captured case also outlives the fix. Once a failure is reproducible, it can join an eval set as a regression case, so the same failure class is checked on every future change. This gives replay a second role beyond debugging: the record built to understand one failure becomes a standing check for its recurrence.