Do LLM judges score markdown-formatted answers higher than plain prose?

Yes, and by far more than any other bias tested: “Judging the Judges” found five judges from four providers favor markdown formatting, headers, bold text, bullet lists, over plain prose making the same point, by 0.10 to 0.76 depending on the judge, against a ceiling of just 0.04 for favoring whichever answer they saw first. Style is the dominant bias the study found, not a minor addition alongside the others.

The practical risk is that formatting becomes a proxy for quality. A judge asked to grade correctness or helpfulness can reward the answer that merely looks more organized, so a genuinely worse answer wins by adding headers and bullets to content a plainer answer states just as well. It’s the same shape as a judge favoring an output that just sounds like its own model family: the judge is rewarding a surface trait correlated with quality instead of quality itself, so calibrating it against human-labeled cases, not just trusting its verdicts, is what catches either one.