Is there a benchmark for whether a coding agent can build a whole agent, not just fix a bug?

Yes: tau-tau-bench (arXiv:2609.04611, pronounced hyper-tau-bench, built by Sierra, the company behind the tau-bench benchmark family it borrows its name from) hands a coding agent a business’s own records, a client’s stated requirements, and a production API, then scores it on whether the complete customer-service agent it builds actually works against simulated users, not on whether a single patch passes a test suite.

Across 53 tasks in four domains, the strongest tested configuration, Claude Opus 5 running under Claude Code, passed only 23.9% of evaluation simulations; an expert-built reference agent scored 82.2%. The paper’s own read of the gap: models write shallow queries instead of understanding the records, say almost nothing to the simulated client, and ship the first design that runs instead of experimenting with architecture or cost. That’s a different failure shape than SWE-bench Pro measures, where the task is landing one fix against a codebase’s own existing tests, not inventing a whole system from a client brief.