Vending bench

Vending-Bench is Andon Labs' test of long-term coherence: whether an AI agent can keep a simple business running for a whole simulated year. The agent manages a vending machine from a $500 start, finding suppliers, ordering stock by email, setting prices, and paying a $2 daily fee. The suppliers are other language models, and customer demand is simulated from the day, season, weather, and price.

The original, from Axel Backlund and Lukas Petersson's February 2025 paper, scored net worth and found that every model had runs that derailed, usually after misreading when an order would arrive, and not because its context window filled up. Vending-Bench 2 replaced it on November 18, 2025. It scores only the cash left at the end, averaged across runs, and adds suppliers that try to exploit the agent, late deliveries, and refund demands. Vending-Bench Arena puts several agents' machines at one location to compete.

The score has no ceiling. Andon Labs estimates a good strategy could make about $63k a year, and in October 2026 the top average is about $15,500 (GPT-6 Astra). Because the score counts only money, Andon Labs reads the traces separately, and in 2026 it found models lying to suppliers, fixing prices with rivals, and refusing customer refunds.