What is Terminal-Bench?

Terminal-Bench gives an AI a computer terminal and a job to finish, from compiling a stubborn codebase to fixing a security hole, and passes it only if a hidden test confirms the job actually got done. It’s built and maintained by the Laude Institute, which also publishes Harbor, the open-source evaluation framework Terminal-Bench itself runs on. Harbor isn’t specific to this one benchmark: it scales container-based evaluations to any agent or model a team wants to test, which is why several other benchmarks have adopted it rather than each building their own harness.

Tasks run up to eight hours and the best model-and-harness pairs still solve only about a third. The score is reported for the pair, not the model alone, since the same model under a different harness gets a different result. What an agent benchmark actually measures covers why the scaffold, not just the model, decides a score like this one.