A Good Judge Can Still Measure the Wrong Thing

TL;DR for operators

An LLM judge can agree closely with human reviewers, produce consistent scores, and still provide insufficient evidence for a product decision. Agreement measures correspondence between evaluation methods; it does not establish that either method adequately captures the quality or risk the decision requires.

LLJ Cards proposes five documentation and evaluation components, containing 45 guiding questions, to address this gap. The framework makes evaluation context, measurement criteria, prototyping, pipeline specifications, and validity evidence explicit.

For organizations using LLM judges to compare model versions, assess safety, or label data, the practical recommendation is to define the intended decision before selecting the judge. Validate the complete measurement procedure against representative and challenging cases, document its configuration, and reassess its suitability when conditions change.

These are methodological recommendations, not experimentally demonstrated improvements in evaluation performance.

Agreement with reviewers is not enough to approve a release

Consider a product team using an LLM to evaluate automatically generated summaries. The judge scores outputs consistently and agrees reasonably well with human reviewers. The team wants to use its scores to compare two model versions and authorize the next release.

What has the agreement established?

It suggests that the automated judge reproduces aspects of the human reviewers’ assessments. It does not establish that those assessments capture every quality dimension relevant to release approval.

For example, a scoring procedure emphasizing readability and information coverage might not adequately penalize a rare but consequential factual omission. Even strong agreement with reviewers would offer limited reassurance if the reference judgments rarely examine such cases.

This distinction becomes important when a score moves from describing outputs to authorizing decisions. A measurement sufficient for comparing average writing quality may not be sufficient for determining whether a system is safe to deploy.

Chehbouni and colleagues address this problem in LLJ Cards: Best practices for the Use of LLMs as Judges.¹ Their proposal shifts attention from the judge’s technical performance alone to the design and justification of the evaluation itself.

Define what the score represents before optimizing the judge

The framework draws on measurement theory, particularly the distinction between a concept, its operational definition, observable indicators, and resulting scores.

Suppose the intended concept is summary quality.

Before an LLM can score that quality, evaluators must decide what it means within the application. Factual fidelity, coverage of relevant information, readability, and suitability for a particular audience are possible dimensions. Their importance depends on the task.

Translating this abstract concept into observable criteria is called operationalization. A judge then applies those criteria to produce scores, which serve as indicators of the intended concept.

The resulting measurement involves several distinct questions:

• Construct validity: Do the criteria and scores adequately represent the quality the evaluation claims to measure?
• Reliability: Does the procedure produce sufficiently consistent results under relevant repeated or changed conditions?
• Convergent validity: Do the scores correspond appropriately with another measurement method, such as human ratings?

Agreement with human reviewers contributes evidence of convergent validity. But if the reviewers and automated judge use similarly incomplete criteria, their agreement cannot establish that the intended construct is adequately represented.

Reliability presents a related limitation. A consistently applied scoring rule can reliably measure a narrow property while remaining inappropriate for a broader decision.

The paper’s central measurement argument is that validity belongs to the justified interpretation and use of scores, not intrinsically to the LLM generating them.

Consequently, a judge validated for one evaluation purpose does not automatically become valid for another.

Five cards make evaluation decisions explicit

LLJ Cards organizes the measurement process into five complementary components. Its 45 questions are methodological prompts, not benchmark observations or experimentally derived acceptance thresholds.

The sequence matters because decisions made early constrain what later tests can establish.

The Context and Evaluation Criteria cards require evaluators to specify the application and measurement target before finalizing a judge. This includes identifying whose interests the criteria represent, what the evaluated cases cover, and which dimensions may be missing.

The LLJ Prototyping card then treats judge development as a documented pilot study rather than an informal search for a satisfactory prompt.

The Pipeline card records the procedure selected after testing. The final Evaluation card asks whether that procedure provides appropriate evidence for its intended use and what consequences may follow from incorrect assessments.

This structure also allows iteration. A pilot revealing systematic failures may require changing the prompt or model, but it may instead expose a problem with the evaluation criteria themselves.

That distinction is essential: improving implementation cannot necessarily repair an inadequately defined measurement target.

Prototype with difficult cases before scaling the evaluation

The paper recommends combining randomly selected evaluation cases with deliberately challenging examples during prototyping.

These two sources of evidence serve different purposes.

Randomly selected cases help examine judge behavior across the sampled evaluation population. Difficult or adversarial cases probe vulnerabilities that ordinary samples may not expose adequately.

For a summary-quality evaluator, a team might include routine summaries alongside examples containing subtle factual distortions, missing qualifications, or conflicting source information. These are illustrative applications of the framework, not experimental cases reported by the authors.

Prototyping should compare candidate models, prompts, and configurations; assess relevant validity and reliability evidence; document refinements; and establish criteria for ending the pilot.

The appropriate stopping rule depends on the evaluation’s purpose. The paper does not provide a universal numerical agreement threshold that makes an LLJ acceptable.

For a team considering full-scale API evaluation, the practical value is the opportunity to identify measurement weaknesses before committing resources to a larger scoring exercise. Whether this process produces net cost savings remains unmeasured.

A reproducible prompt is not a reproducible evaluation

Once a configuration is selected, the LLJ Pipeline card requires more than preserving the final prompt.

The recommended record includes the judge’s model family and version, access date, sampling parameters, evaluation configuration, prompt structure, score computation, post-processing, implementation availability, and resource costs.

These details determine what procedure actually produced the reported measurements.

Two evaluations using identical prompts may differ because the underlying models, inference settings, comparison formats, or score aggregation procedures differ.

Documentation makes such differences inspectable. It does not guarantee exact replication, particularly when proprietary model providers update or retire previously available systems.

The reproducibility claim must therefore remain specific: complete reporting supports examination and potential reproduction of the procedure, subject to the availability of its dependencies.

Validate the decision that uses the score

The LLJ Evaluation card extends validation beyond a single aggregate agreement metric.

It calls for appropriate benchmarks and baselines, disaggregated assessment, explicit limitations, and consideration of ethical and social consequences.

This changes how a product organization should interpret evaluation evidence.

A judge’s aggregate performance may be acceptable while errors concentrate in particular content categories or stakeholder populations. Those differences matter when the scores determine which outputs are approved, rejected, or escalated.

The paper also emphasizes that score validity does not automatically transfer across tasks, domains, or deployment conditions.

A judge initially assessed on general summaries should therefore not be assumed suitable for high-stakes specialized summaries without additional evidence.

Cognaptus inference: organizations can translate this principle into use-specific evaluation authorization.

A judge configuration might be approved for comparing average response quality within a defined dataset while remaining unauthorized to make individual safety decisions. Such restrictions should follow from the tested criteria, relevant failure modes, and consequences of incorrect scores.

When the application, evaluated population, scoring criteria, or underlying judge changes materially, the supporting validity argument should be reconsidered.

The framework motivates this form of conditional governance, although it does not prescribe an organizational approval system or establish an optimal division of responsibility between human and automated evaluators.

The framework is proposed, not experimentally validated

The evidential boundary is significant.

LLJ Cards is a literature-grounded methodological synthesis. Its development draws on measurement theory, natural language generation evaluation, human annotation practices, and responsible AI documentation.

The authors motivate the framework through three application areas: text summarization, safety alignment, and data labeling.

They do not report a controlled experiment testing whether adopting the cards improves judge accuracy, reliability, reproducibility, practitioner decisions, or organizational outcomes.

The supporting literature synthesis is also not presented as a systematic review with a reproducible search and study-selection protocol. Applicability beyond the motivating domains remains open.

Accordingly, the paper supports the rationale and structure of a more disciplined evaluation process, but not a quantified claim about its effectiveness.

Organizations adopting the framework would still need to test whether the additional documentation and validation effort improves their evaluation decisions sufficiently to justify its costs.

Treat judge validity as permission for a particular use

The central contribution of LLJ Cards is a clearer separation between producing scores and justifying reliance on them.

A capable judge can implement a measurement procedure consistently. It cannot, through benchmark performance or prompt optimization alone, establish that the procedure measures the appropriate construct or supports every decision made from its results.

For operators, the practical sequence is to define the intended decision, specify the measurement target, test the proposed procedure, preserve its implementation details, and evaluate the evidence supporting its use.

The question is no longer simply whether an LLM judge agrees with human evaluators. It is whether the available evidence justifies trusting its scores for this particular decision, under these particular conditions.

That standard is the methodological proposition of LLJ Cards. Its measurable effectiveness remains a question for subsequent research and implementation.

Cognaptus: Automate the Present, Incubate the Future.