Different Rubrics, Nearly the Same Scoring Lever

TL;DR for operators

An AI evaluator may report separate judgments about grammar, vocabulary, and coherence without applying those criteria independently. In a study of 25 base language models, interventions targeting the rubric that had been degraded produced only 0.0074 more expected score points than interventions targeting other rubrics—just 1.74% of the absolute matching effect. The difference was statistically detectable but far below the researchers’ practical significance threshold.

The models nevertheless contained recoverable scoring information, and several rubric-related representations were distinguishable. The failure concerned selective control, not an inability to represent or change scores.

For teams procuring or developing AI assessment systems, aggregate agreement with human judgments should therefore be supplemented by tests of criterion-specific responsiveness. The study does not establish that deployed instruction-tuned or fine-tuned evaluators share the same weakness.

A rubric should respond to the criterion it measures

Consider an automated essay scorer that reports judgments about grammar, vocabulary, and cohesion. A student submits a revised essay with better grammar but unchanged vocabulary and organization. An independently functioning assessment system should reflect that improvement primarily in its grammar judgment.

The same requirement applies beyond education. A model evaluating customer-support responses for factual accuracy, completeness, and tone should distinguish which criterion has changed when one aspect of the response is deliberately altered.

This creates a verification problem. A model can produce credible criterion labels, and its overall scores can correlate with human judgments, while the internal mechanisms producing those scores remain coupled.

Huang, Wang, and Hsieh examine this problem in Inside the Scorer: Mechanistic Evidence on Rubric Control in LLM-Based Automated Essay Scoring from 25 Base Models.¹ Their central finding is that interventions associated with different writing rubrics often affect the model’s score in nearly interchangeable ways.

The distinction matters because an evaluator’s ability to describe a criterion does not demonstrate that the criterion independently governs its decision.

Three tests separate representation from control

The researchers studied 25 open-weight, non-instruction-tuned language models across 14 families, using 75 human-rated responses from Taiwan’s BESTEP English writing assessment. The assessment covers short factual answers, opinion emails, and integrated essays, with six analytic rubrics applied in different combinations.

Their audit examines three increasingly demanding properties.

First, can internal representations predict writing quality? The researchers extract model activations and train ridge regression probes to predict human holistic band scores. A successful probe establishes that score-related information is recoverable from the model’s internal state. It does not establish that the model uses that information when producing a score.

Second, are rubric representations distinguishable? Starting with 26 high-band essays, the researchers generate 104 deliberately degraded versions targeting particular rubric criteria. Two trained graders verify that the intended degradations are recognizable. The activation difference between an original and its degraded version defines a rubric-associated direction in representation space.

Third, does intervening along that direction selectively change scoring behavior? The researchers inject rubric directions into model activations and measure changes in the model’s expected numeric band score. They compare interventions matching the degraded criterion against interventions targeting other criteria, under paired experimental conditions.

This final test provides causal evidence about the interventions’ effects on the model’s overall expected score. It does not directly measure changes in separately reported human-validated rubric subscores.

The models distinguish rubrics better than they control them

The first two tests produce substantial positive results, particularly for longer responses.

Sources: Section 4.1, Table 3; Section 4.2, Table 4. Geometry comparisons retain only sufficiently reliable activation directions.

The positive separation gaps for longer responses mean that directions associated with the same rubric tend to be more similar than directions associated with different rubrics. This suggests some meaningful organization of rubric information inside the models.

Yet representational differences vary considerably. Grammar and spelling/punctuation directions exhibit extremely high noise-corrected similarity, whereas argument relevance and grammar are far more distinct.

These observations establish neither the complete independence of rubric representations nor their causal role in scoring.

That becomes clear in the intervention results.

Across 74,304 paired steering cells, the model-aggregated advantage of matching over mismatched rubric steering is:

• 0.0074 band-score points, with a 95% confidence interval of 0.0036–0.0117.
• 1.74% of the absolute matching effect, with a 95% confidence interval of 0.85%–2.74%.

The confidence interval excludes zero, but the magnitude is small relative to the researchers’ declared practical threshold of 0.05 points or 10% of the matching effect.

Importantly, steering itself often works. Five of six rubrics show positive matching-direction effects, ranging from 0.168 to 0.359 expected score points. Argument relevance is the exception, with no detectable positive matching effect.

Thus, the primary failure is not that the interventions cannot move scores. It is that choosing the direction associated with the degraded rubric adds very little benefit over choosing another rubric’s direction.

Among four reliably measured, effectively steered rubrics—grammar, vocabulary, cohesion, and spelling/punctuation—none of the ten testable ordered head-to-head comparisons shows a detectable advantage favoring the matching direction. Individual intervals can be wide, however, so these comparisons do not prove exact equality.

A shared component explains much of the score movement

One possible explanation is that different rubric directions contain a common component associated with generally higher essay quality.

The researchers investigate this through additional intervention controls.

When they remove the shared axis across rubric directions, positive steering effects largely disappear. Corrected matching and mismatched effects average approximately −0.093 and −0.100 points, respectively, leaving a matching advantage of only 0.0064 points.

This is a mechanism-focused diagnostic: removing common information removes most of the positive score-raising behavior.

A more targeted control removes overlap between pairs of rubric directions. The remaining components still influence scores more than norm-matched random directions, suggesting that the residual representations are not simply meaningless noise. Nevertheless, the matching advantage is only 0.013 points, with a 95% confidence interval of −0.015 to 0.042.

The researchers also examine whether comparatively better scorers exhibit stronger independence. Among seven models whose overall scores correlate with human ratings at 0.6 or above, the matching advantage remains 0.011 points, approximately 1.9% of the matching effect. This subgroup analysis is exploratory rather than confirmatory.

Together, these controls favor an interpretation of predominantly shared score control, with some residual rubric-associated information that does not yield consistently selective interventions.

They do not establish that the model contains no distinct rubric information.

Procurement tests should challenge each criterion separately

Cognaptus inference: When independent criterion-level accountability is a product requirement, an evaluation team should demand evidence beyond overall scoring accuracy and plausible criterion explanations.

A useful acceptance protocol would deliberately modify one assessment criterion while preserving others as closely as possible. It would then measure whether the intended criterion responds more strongly than unrelated criteria, using matched and mismatched controls.

For example, an education technology provider evaluating an automated writing scorer could test grammar-targeted changes across multiple essay types, measure the resulting criterion and overall scores, and repeat the exercise with vocabulary- and cohesion-targeted changes. Reliability checks would establish whether the perturbations genuinely isolate the intended quality dimension.

These tests can be adapted to black-box products through controlled inputs and observable score changes. The paper’s internal activation interventions offer a more direct mechanism audit when model access permits, but its numerical results do not validate a particular black-box acceptance protocol.

Where independent criteria are mandatory, explicitly separated scoring components or constrained criterion contributions may be architectural alternatives worth testing. Neither architecture was validated by this experiment.

The procurement question consequently becomes more precise: Can the supplier demonstrate that each declared criterion responds preferentially to changes in that criterion, across the tasks for which the system will be used?

The finding has clear experimental boundaries

The study evaluates base models up to 10.31 billion parameters, not instruction-tuned, chat-tuned, or task-specific fine-tuned assessment systems. All models use a common grading prompt, and the causal interventions add activation directions at one layer. Other interventions or trained scoring architectures could behave differently.

The original assessment dataset contains only 75 responses, while the 104 degraded versions originate from 26 essays. Verification confirms recognizable targeted degradation, not perfect isolation of every other rubric dimension.

A further identification problem remains unresolved. The dataset provides holistic human ratings, not individual human rubric scores. Consequently, the shared model control might reflect internal compression of distinct criteria, correlations or halo effects in human judgments, or some combination. The experiment cannot distinguish these explanations.

These limitations restrict generalization without erasing the observed mismatch between rubric representation and selective intervention.

Independent criteria require independent evidence

The study establishes a specific failure of inference: recovering rubric-related information from a model, or observing that a rubric-associated intervention changes its score, is insufficient to demonstrate independent criterion control.

For organizations using AI to evaluate writing, work quality, compliance, or other multidimensional outputs, that finding changes the evidence required for acceptance. A criterion label identifies what a system claims to evaluate. Targeted responsiveness tests are needed to establish whether the claim corresponds to a functioning decision mechanism.

Cognaptus: Automate the Present, Incubate the Future.