What can replay not tell me?

Replay can’t tell you why the model made the choice it made, only whether a given input and context reliably produce a given output. You can vary one condition and watch the output move, which narrows down what mattered, but the model’s actual reasoning, the internal weighing that got it there, stays outside what you observed. Replay also can’t tell you how common the failure is: one replayed case is one case, not a rate, so it doesn’t say whether this happens on one percent of sessions or one in a million. And replay is only as honest as the capture. If the original trace didn’t record a piece of context, a tool argument, a system prompt version, replay can’t recover it, and you’re debugging an incomplete picture without knowing it’s incomplete.