Agent benchmarks

An agent benchmark is a fixed set of tasks, an environment to run them in, and a scoring rule, used to compare models and agent designs on something repeatable. There are dozens now, and they sort into families by what the agent has to do.

Tool-calling benchmarks check whether the model picks the right function, fills the arguments correctly, and calls nothing when nothing fits. Conversational benchmarks put the agent in a dialogue with a simulated user and a policy, and score whether the final state is right. Coding benchmarks hand it a real repository issue and run the tests. Terminal and computer-use benchmarks give it a shell or a screen and a task to finish. Web benchmarks drive a browser. General assistant benchmarks mix browsing, files, and several steps, and the newest ones run asynchronously so events happen while the agent works. Multi-agent benchmarks score a team rather than one model. And a separate family benchmarks the judges themselves.

Read any score with three things in mind. First, the scoring rule: most grade the end state, a few grade the path, and only some run each task more than once to check the agent succeeds every time rather than sometimes. Second, the scaffold: the score belongs to the prompts, tools, and retries wrapped around the model in that run, not to the model alone, and rarely to your setup. Third, the age: public tasks leak into training data, which is why benchmarks keep releasing harder, held-out versions, and why top scores drop sharply when they do.

None of this predicts how reliably your agent runs in production. The tasks are curated, and the inputs that break agents are the ones nobody wrote down. Benchmarks are for choosing between models. Your own traces are for knowing whether the agent works.