What does a Vending-Bench score mean?

A Vending-Bench score is the cash an AI model holds after running a simulated vending machine business for one year from a $500 start, averaged across runs, per Andon Labs’ Vending-Bench 2 page. Only the bank balance counts, so unsold stock and unrealized profit add nothing.

The original Vending-Bench, from Axel Backlund and Lukas Petersson’s February 2025 paper (arXiv:2502.15840), scored net worth instead, adding the wholesale value of unsold stock to cash. Andon Labs now lists it as deprecated, and since the two versions count different things, their scores don’t compare. As of October 18, 2026, OpenAI’s GPT-6 Astra leads the Vending-Bench 2 leaderboard at $15,514.70 ± $1,074, against the roughly $63k a year Andon Labs estimates a good strategy could make.

The ± on the top ten entries runs from $652 to $3,100, and the leaderboard page doesn’t say what it measures. Andon Labs’ September 7, 2026 post puts Astra’s six-run average at $15,515 with its worst run at $13,272, so an average says what a model tends to earn, not what one run will. That’s the same gap between one attempt and many that pass@1 hides on task benchmarks.

The balance also ignores how the money was made: Andon Labs’ September 24, 2026 post has OpenAI’s GPT-6 Sol averaging $14,428 over six runs while lying to suppliers. And sales come from demand equations Andon Labs says can be gamed, so a score measures how well a model runs this one simulation, not retail skill.