Can an OSWorld task fail because of a bug in the benchmark, not the agent?

Yes. OSWorld 2.1’s own release notes name a task, Task029, whose unchanged setup can report a 120-second HTTP-server startup timeout that has nothing to do with what the agent did; the notes say a fix for it was tested separately but didn’t ship in this release.

The same release notes limit how far to trust a clean run, too. Validation for 2.1 covered 326 automated tests and hash checks on all 108 task files, plus a check of Docker artifact integrity and runtime references, but a live Docker VM boot was not tested. That’s the step that actually runs a task end to end, so a release can pass every listed check and still carry an environment bug a boot would have caught.

Both cut the same way: a raw pass or fail on OSWorld can be the environment’s result, not the agent’s. Read a release’s known-issues list before treating a single task’s outcome, in either direction, as a verdict on the agent.