A Good Judge Can Still Measure the Wrong Thing
TL;DR for operators An LLM judge can agree closely with human reviewers, produce consistent scores, and still provide insufficient evidence for a product decision. Agreement measures correspondence between evaluation methods; it does not establish that either method adequately captures the quality or risk the decision requires. LLJ Cards proposes five documentation and evaluation components, containing 45 guiding questions, to address this gap. The framework makes evaluation context, measurement criteria, prototyping, pipeline specifications, and validity evidence explicit. ...