Terminal bench

Terminal-Bench gives an AI a computer terminal and a job to finish. The current version, 4.0, came out in August 2026. Each task is a written instruction, a container with the right software installed, and a hidden test that checks whether the job got done. Tasks range from compiling a stubborn codebase to fixing a security hole to training a small model. Version 3.0 had 74 tasks across seven domains; 4.0 removed eight that every frontier model had started solving every time.

A task passes if its test passes at the end. The score is the share of tasks passed, and it's reported for a pair: a model plus the agent harness driving it, such as Claude Code or Codex. The same model in a different harness gets a different score, which is a useful reminder that the harness is part of the product.

It's hard because tasks are long. Agents get up to eight hours per task and still fail most of them. The best pairs solve about a third. What the score doesn't tell you is how the agent behaved on the way: a run that quietly deleted the wrong directory and then recovered counts the same as a clean run.

Each version raises the bar as the previous one saturates. Terminal-Bench 2.0, from November 2025, had 89 tasks and its launch leader solved about half. 3.0 rebuilt the set for harder, longer work. 4.0 tuned time and memory limits per task and dropped the ones that no longer separate models.