What is an agent benchmark?

An agent benchmark is a fixed set of tasks, an environment to run them in, and a scoring rule, used to compare models and agent designs on something repeatable. They sort by what the agent has to do: tool-calling benchmarks check whether it picks the right function, conversational benchmarks score a dialogue against a simulated user, coding benchmarks hand it a real repository issue and run the tests, and terminal and web benchmarks give it an environment to operate in directly. One family scores a whole team of agents, and another benchmarks the judges instead.

A benchmark score does not transfer to your agent, because the benchmark fixes the scaffold and you don’t get to keep it. Every model on a leaderboard runs behind the same prompts, retries, and tool wrappers the benchmark authors wrote; swap in your own prompt, retries, and tool descriptions, and two tied models can trade places on your task. The benchmark also never sees your inputs: it replays a fixed task set, not your agent’s traffic. A high score says a model can do this task family under this scaffold, nothing about yours. Production reliability is measured on your own traces for that reason.