Which graders belong on every PR and which belong on a nightly run?

Cheap, deterministic graders belong on every PR; anything that calls a language model belongs on a scheduled run instead. A PR gate fires on every commit from every contributor, so whatever runs there multiplies by your team’s commit volume, and a rule-based check or a distilled classifier costs close to nothing at that frequency.

An LLM judge doesn’t. It’s a paid inference call, and running one on every PR means paying for it at commit frequency instead of once a day. Moving the judge to a nightly run collapses that multiplier: it grades a fixed batch once instead of once per commit. The tradeoff is latency, not accuracy, since a nightly run catches what a PR gate would have caught, just hours later. Reserve the PR tier for checks cheap enough to run at commit frequency, and let volume decide the rest.