Can an agent score well on every check and still fail the task?

Yes. A 2026 benchmark of 842 sandboxed, multi-requirement tasks found rubric pass rates between 70.0% and 85.7% across eight agents when each requirement is scored on its own, on average, but between 12.6% and 40.9% when a task counts as done only if every one of its requirements holds at once. The best agent tested, Opus 4.8, led the strict count at 40.9%, well under the rubric-average range that makes the same agents look strong.

The gap is arithmetic, not a different capability. A task with five independent requirements, each met 90% of the time, clears all five together only about 59% of the time, and real tasks compound more requirements than that. SkillForgeBench builds its tasks from the same skill specification an agent would follow to do the work, then scores both ways on purpose: a rubric average for partial credit, and a strict pass for whether the task is actually finished. Pass^k, the metric tau-bench uses to require every repeat of a task to succeed, catches the same kind of gap from a different angle, run count instead of requirement count.

A benchmark that reports only the average is reporting the easier number.