Does Terminal-Bench's own test data have grading errors?

Yes. Epoch AI’s independent review of Terminal-Bench 4.0 found scoring defects in 30 of its 66 tasks, 45.5%, and rated the cleaned-up benchmark “Flawed.”

The 4.0 release was itself a cleanup: it dropped 8 of the prior version’s tasks, two each for saturation, model refusals, publicly available solutions, and unresolved quality or platform problems, fixed 19 more, and moved every task to a flat 8-hour agent timeout. Epoch’s audit found the cleanup didn’t go far enough. Nineteen of the 66 tasks are false positives, where a submission can pass without really solving the task, through reward hacking (reading the pass signal straight out of the grading container) or answer leakage (the correct answer is readable or derivable inside the container). Eleven are false negatives, where an undocumented requirement fails a plausibly correct answer.

A score on Terminal-Bench 4.0 still mixes real capability with grading noise: a leaderboard jump can mean a model found a leak rather than did harder work, and a drop can be the benchmark’s fault rather than the agent’s.