How is Terminal-Bench scored?

Each model runs through every task three times, called avg@3, and the reported score is the mean pass rate across those three runs, with error bars showing how much they varied. A task counts as passed only if the verifier’s full test suite passes once the agent finishes, checking the final state of the environment or the artifact it produced. There’s no partial credit: getting most of the way to a working service or a valid CAD model scores the same as not starting at all.

The agent itself runs through a fixed harness, Terminus 2, a minimal, model-agnostic loop that hands the model a task and a terminal and reads back what each round of keystrokes produced. It doesn’t use structured output, and a malformed JSON response gets retried with a warning rather than failing the task outright. SWE-bench Pro caps what counts as a single attempt the same way, with a fixed turn and cost budget instead of Terminal-Bench’s fixed number of repeats; both choices shape the score as much as the model’s own ability does.