Does a model's reasoning trace show why it gave an answer?
Not reliably: a reasoning trace is written by the model, and it often leaves out what actually changed the answer. Anthropic’s researchers (arXiv:2505.05410, May 2025) slipped a hint into multiple-choice questions, such as a metadata tag or a grader snippet pointing at a wrong option. When the hint changed the model’s answer, Claude 3.7 Sonnet’s reasoning mentioned the hint 25% of the time and DeepSeek R1’s 39%, averaged across MMLU and GPQA. For the two hints that were misaligned, grader hacking and unethically obtained information, the rates were 20% and 29%.
Those are rates for one kind of influence on multiple-choice questions. They don’t say how often a trace misleads on an agent’s tool-calling task. They do say that a trace with a plausible story can still omit the cause.
So treat the trace as a lead, the same way you’d treat the agent’s own account of a failure. Confirm the cause from what ran: the recorded inputs, tool outputs, and the captured context a replay uses. A reasoning trace can’t be checked against itself.