Is there a benchmark that tests an agent against real MCP servers instead of mocks?

Yes. MCP-Atlas runs 1,000 human-written tasks across 36 real, production MCP servers and 220 tools, none of them mocked, with prompts that never name the server, tool, or parameters an agent needs, so it has to find the right tool among plausible distractors first.

Scoring 20 frontier models against roughly 6,900 diagnosed failures, the authors split errors into tool-call failures and what they call cognitive failures: misreading the task, stopping before gathering enough evidence, or misreading the tool output once it’s back. Cognitive failures account for 63.3% of the total, tool-call failures the rest. Weaker models skew the other way: o3 Pro fails on the tool call itself 57.6% of the time, with 40.1% of its failed runs never invoking a tool at all despite the task requiring one.

That’s MCP-Atlas’s own finding, not a claim about tool-calling evals generally: against real servers, once a model clears some baseline competence, what it does with a tool’s output decides more than whether it picked the right tool, one of the three places a tool call can fail.