What are the main types of agent benchmarks?
Six, split by what the agent is asked to do: pick the right tool call, hold a conversation against a policy, fix a real bug, operate inside a changing environment, build something as a team, or score how well a judge model grades others.
Tool-calling benchmarks like BFCL check whether a model picks the right function and fills in the right arguments. Conversational benchmarks like tau-bench run a simulated user against a policy document and score only the account changes that resulted. Coding benchmarks like SWE-bench Pro hand the agent a real bug report and grade it against the project’s own tests, while environment benchmarks like Terminal-Bench and Gaia2 drop it into a shell to finish a job, or a simulated phone where time keeps passing, and check what state it leaves behind. The last two score different units: one grades a whole team of agents building something end to end, and the other benchmarks the judges, where a judge’s own rank can shift by up to 14 positions from one benchmark to the next.
A tool-calling score doesn’t predict how an agent holds a conversation, and neither predicts how it behaves on your own traffic.
See also: What is OSWorld?