Does a more expensive model make an agent more reliable?

Not in ReliabilityBench, where the model that cost 82 times more was no more reliable. GPT-4o scored 90.42% on the benchmark’s pass-both-of-two-tries measure and Gemini 2.0 Flash 91.04%. On clean tasks they scored 95.00% and 96.88%, and on heavily reworded tasks 88.75% and 88.12%.

The bill did differ: $9.77 for GPT-4o’s 480 episodes against $0.12 for Gemini’s, at the paper’s December 2024 list prices of $2.50 and $10.00 per million input and output tokens for GPT-4o, and $0.075 and $0.30 for Gemini.

Price didn’t buy reliability for these two models on these tasks, though routing to a cheaper model can still cause a silent failure in other setups. Measuring your own agent’s consistency settles it for your tasks. The test itself covers two models and 20 tasks, and the author lists broader model coverage as future work, so it can’t say a bigger model never helps.

sources

keep reading

More on this.

Does adding self-reflection to an agent make it more reliable?Not in ReliabilityBench: a plain ReAct agent beat the same agent with Reflexion self-reflection added, on reworded tasks and when its tools failed.answer →Does an agent that falls for an injected fault in a benchmark ignore available evidence?Not necessarily: in one study, of the runs where contradicting evidence eventually appeared, only 17 of 44 had it arrive before the agent already acted on the fault.answer →How do you measure whether an agent succeeds consistently, not just once?Run the same task repeatedly and score whether every run passes, not whether any one does; ReliabilityBench calls this metric pass^k.answer →Everything on reliabilitybenchThe definition, and every question answered under reliabilitybench.concept →

Try Tessary on Cloud.

start for freeor self-host

free · 10,000 traces a month · 1 GB · 30-day retention · $10 one-time credit for triage and RCA · no credit card