Rules of Attraction: How LLMs Learn to Judge Better Than We Do

Rubrics are supposed to make judgment boring.

That is their charm. A good rubric tells a teacher why one essay deserves a 5 instead of a 3, tells a compliance reviewer why one response is acceptable and another is risky, and tells an internal QA team why a generated summary is useful rather than merely confident. In business, boring judgment is valuable. It scales. It can be audited. It survives employee turnover. It does not wake up one morning and decide that “clarity” now means “vibes with a semicolon.”

LLM evaluators, unfortunately, are very good at sounding like they have rubrics and much less reliable at actually following them. The paper Learned-Rule-Augmented Large Language Model Evaluators tackles that uncomfortable gap directly.¹ Its proposal is not simply “use a bigger judge model” or “ask the model to think step by step with more ceremony.” The authors argue that evaluator failure often begins earlier: the model is judging with unstable standards, and even when standards are supplied, it may not apply them consistently.

That distinction matters. If the problem were only model intelligence, the solution would be straightforward: buy more parameters, wait for the next reasoning model, pretend the invoice is strategy. But the paper points to a more operational idea: evaluation quality depends on rules, rule-data alignment, and rule-following. Bigger models may help. They do not remove the need for law.

The paper’s contribution is a learned-rule evaluator pipeline. First, it uses LLM-assisted Monte Carlo Tree Search, or MCTS, to distill task-specific scoring rules from labeled data. Then it applies those rules in two ways: Chain-of-Rule prompting, or CoR, which injects learned rules into prompts without training; and a Rule-Augmented Evaluator, or RuAE, which trains a 7B model with reinforcement learning so that it applies the rules more consistently.

The interesting part is not that rules help. We already knew that, in the same way we know seatbelts help. The interesting part is how the paper tries to make rules learnable, portable, and executable by models that otherwise enjoy improvising standards with great confidence.

The real failure is not bad judgment; it is moving judgment

Most discussions of LLM-as-a-judge systems focus on whether model outputs correlate with human preferences. That is useful, but it hides two different failure modes.

The paper names them as two misalignments. The first is a mismatch between evaluation principles and human-labeled data. A handcrafted rubric may sound reasonable but still fail to match how labeled examples are actually scored. The second is a mismatch between the LLM’s understanding of the principles and its application of those principles. In plain English: the rubric may be wrong, and even if it is right, the model may still wander off-script.

The authors motivate this with an exploratory study on the ASAP essay-scoring dataset. They prompt Qwen-7B to propose evaluation principles and score essays, then analyze 600 responses. The resulting principles are dispersed rather than unified. Even within the same evaluation dimension, consistent scoring remains difficult. That is not a minor prompt-design inconvenience. It means the model is not merely producing different answers; it is implicitly switching standards.

This is where the usual “just use CoT” habit becomes suspicious. Chain-of-thought can make an evaluator more verbose, and occasionally more transparent, but it does not guarantee that the evaluator is using a stable scoring frame. A model can produce a beautiful rationale while grading from the wrong constitution. Very elegant. Also useless.

The paper’s replacement idea is simple: instead of asking the model to invent evaluation principles at inference time, learn scoring rules from data first. Then use those rules to constrain judgment.

MCTS searches rules, not token-level thoughts

The first stage is rule distillation. A scoring rule is represented as a set of sub-rules. Each sub-rule contains an evaluation aspect and a rubric. For essay scoring, aspects might include organization, word choice, idea and content, sentence fluency, and evidence support. For scientific document relevance, aspects might include application, findings, goal, domain, and idea.

The important design choice is that MCTS does not search token-level reasoning paths. Token-level MCTS is painfully large because every next word creates another branch. This paper searches at the level of rules. A state is a candidate rule set. An action either adds a new sub-rule or modifies an existing rubric. Modification is deliberately restricted to stricter or more lenient versions. That restriction narrows the search space, though it also limits the kinds of rule revision the system can discover.

The MCTS process has two phases:

Each candidate rule set is evaluated by using an LLM as the scoring environment and comparing its predictions against ground-truth labels with task-specific metrics. For regression-style tasks, this may involve error measures; for ranking tasks, ranking metrics matter. To reduce cost, the method treats independent sub-rules as equally weighted and caches prior sub-rule predictions. That is not glamorous, but it is exactly the kind of engineering assumption that makes a research idea less decorative and more runnable.

The mechanism also clarifies why this is not just prompt engineering in formal clothing. A normal rubric prompt begins with human judgment. This method begins with labeled data and searches for interpretable rules that make model scoring better match that data. The rule is not merely written. It is selected.

CoR is the cheap fix; RuAE is the trained judge

Once the system has learned rules, the paper tries two execution strategies.

The first is Chain-of-Rule. CoR filters out low-frequency rules, selects the top five high-value rules by average reward, samples from them, and inserts the selected rule into the evaluation prompt. It is training-free. That makes it attractive for organizations that cannot or should not fine-tune a model for every evaluation workflow.

CoR is the practical baseline that many businesses would try first. It says: keep your existing LLM, but stop asking it to judge from a vague instruction. Give it learned criteria.

The second strategy, RuAE, is heavier. The authors train a rule-augmented evaluator using reinforcement learning based on Qwen2.5-7B-Instruct. The reward has two components. One rewards absolute score accuracy: how close the predicted scores are to ground truth. The other rewards order preservation: whether the model correctly distinguishes the relative quality of paired texts.

That second component is important. In many evaluation systems, the exact score is less important than the relative decision. Which résumé should be reviewed first? Which generated answer is safer? Which paper is more relevant? Which complaint deserves escalation? A model that predicts scores with plausible decimals but gets pairwise ordering wrong is a spreadsheet with delusions.

The paper trains RuAE with Group Relative Policy Optimization, using pairwise prompts where the model selects important aspects, assigns weights, evaluates both texts, and produces scores. The implementation is not cheap: one epoch, maximum prompt and response lengths of 2048 tokens, vLLM rollout, bfloat16 precision, and 5×A800 80GB GPUs. This is not “copy this prompt into your SaaS dashboard and call it governance.” It is a serious training procedure.

The results favor rules most when evaluation is cognitively heavy

The experiments cover four task types: automated essay scoring with ASAP, scientific document relevance with Relish, product review rating prediction with Amazon reviews, and summarization meta-evaluation with SummEval. The baselines include vanilla scoring, CoT-style scoring, large reasoning models such as DeepSeek-R1 and QwQ-32B, task-specific models, and supervised fine-tuning of Qwen-7B.

The pattern is not “rules magically win everywhere.” The pattern is more useful than that: rules help most when the evaluation requires complex criteria, long-form analysis, or stable multi-aspect judgment.

A few reported results make the mechanism visible:

On ASAP, the contrast is especially sharp. Qwen-7B vanilla scoring reports 0.286 QWK, CoT drops to 0.122, CoR improves to 0.316, and RuAE reaches 0.379. DeepSeek-R1, a much larger reasoning model, scores 0.315. The paper reports RuAE as 20.3% above the second-best model on ASAP QWK.

The interpretation should be careful. This does not mean a 7B evaluator is generally superior to a giant reasoning model. It means that for this task, with learned rules and RL alignment, a smaller model can become a better judge than a larger model operating with less task-aligned structure. In other words, architecture plus discipline can beat raw eloquence.

Relish shows a similar but slightly more nuanced picture. RuAE is strongest on nDCG, reaching 0.934, while mAP remains competitive but not universally best across all methods. That distinction matters because ranking metrics capture different aspects of retrieval quality. A business reader should not convert “strong Relish performance” into “dominates all retrieval evaluation metrics.” That would be convenient. It would also be lazy.

Amazon is the useful spoiler. RuAE does not dominate the review-star task. The paper suggests that Amazon’s short-text rating prediction may demand less advanced reasoning, making the benefits of rule-following less pronounced. This is exactly the kind of negative texture that makes a result believable. If rules helped equally everywhere, the correct response would be to check whether the evaluation setup had accidentally become a marketing brochure.

SummEval is also mixed. CoR performs strongly, especially in the Qwen-32B family, but RuAE is not the overall winner. The authors speculate that the Qwen-32B family may already have task-specific advantages. That is a reminder that evaluator performance is not only about method; it also depends on what a base model already knows, or has already seen.

The ablations say RL is not just expensive decoration

The paper’s ablation studies are where the mechanism becomes more convincing.

The framework ablation tests three variants: removing the order component from the reward, using MCTS-generated trajectories for supervised fine-tuning, and using MCTS only, equivalent to rule-based prompting without model training. The variants generally reduce performance. MCTS+SFT suffers the largest drop. The authors explain that high-reward data often focuses on easily evaluable samples, introducing bias, and that training examples may contain inaccurate scoring in specific dimensions even when the final result is correct.

That is a useful warning. Supervised fine-tuning can teach a model to imitate answer patterns. It does not necessarily teach the model to preserve the evaluation logic that made those answers good. In evaluator systems, that difference is expensive. A model that imitates the surface of a good judgment may still fail when the case gets slightly harder, longer, or less typical.

The reward ablation also has a meaningful interpretation. Removing the order-preservation reward has little impact on Amazon but hurts ASAP more noticeably. That matches the task structure. Review-star prediction is closer to independent ordinal regression; essay scoring involves stronger relative judgments and broader scoring standards. For businesses, the lesson is that reward design should reflect the decision being automated. If the business cares about ranking candidates, preserving order is not a technical flourish. It is the task.

The rule-distillation ablation compares the authors’ dataset-level reward computation with a pairwise reward alternative. The pairwise alternative produces higher entropy and lower Jaccard similarity among top rules: $H = 3.791$ and $JS = 0.127$, compared with the authors’ $H = 2.960$ and $JS = 0.467$. The paper interprets this as evidence that the authors’ method identifies more stable and unified rules.

That finding is not merely statistical housekeeping. Stable rules are what make evaluation auditable. If the rule search produces a different rubric every Tuesday, the organization has not built governance. It has built a slot machine with YAML.

The learned rules are interpretable enough to inspect

One attractive feature of the paper is that the learned rules are readable. The model does not produce only hidden weights or opaque preference scores. It produces candidate dimensions and rubrics that humans can inspect.

For ASAP, the distilled rules recover five of six human-defined rule categories: evidence and support, organization, word choice, sentence fluency, and ideas and content. The missing category is conventions. The reported alignment metrics are strong: precision 1.00, recall 0.83, Jaccard similarity 0.83, and hypergeometric test $p = 0.024$. The paper also reports performance above a random benchmark of 1.67, described as a 66.7% improvement over random selection.

This is important because business evaluation systems need more than accuracy. They need explainability that is not theater. A compliance evaluator that says “score: 2.7” is not enough. A useful evaluator should tell a reviewer which criteria were considered, what each criterion meant, and why the final score followed from those criteria.

The paper’s score-distribution analysis adds another layer. On ASAP, vanilla scoring overpredicts scores of 2 and 3 and underestimates the more common score of 4. CoR improves the distribution but remains concentrated around 3 and 4. RuAE aligns more closely with the ground truth. This analysis is best understood as calibration evidence, not a separate grand thesis. It shows that learned rules and RL do not merely improve a headline metric; they also reduce distributional bias in the scores.

The robustness test on sub-rule generation is also modest but relevant. The authors repeat candidate rule generation 15 times at higher temperature and measure Jaccard similarity. Most rules fall around 0.3 to 0.4 similarity, with a global mean shown as 0.33 in the figure, and some rules closely resembling many others. This is a robustness or sensitivity check: it supports the claim that rule generation is not wildly unstable under similar task prerequisites. It does not prove universal robustness across domains where evaluation criteria are inherently contested.

Business translation: build an evaluator governance layer, not another judge prompt

The paper directly shows that learned rules can improve LLM evaluation across several tasks, especially where judgment is complex and multi-aspect. It also shows that a smaller RL-trained evaluator can outperform much larger models on selected complex evaluation metrics.

Cognaptus’ business inference is broader but should be kept disciplined: organizations should treat evaluation as a governance layer, not as a prompt attached to the end of a workflow.

A practical implementation would look something like this:

This has immediate relevance for automated essay scoring, quality assurance of generated content, review ranking, document relevance, customer support auditing, policy-compliance screening, and internal knowledge-base evaluation. Anywhere a company asks an LLM to “score this,” the natural next question should be: score according to what rule, learned from which data, and enforced how?

The cheap version is CoR. It may be enough for workflows where the cost of occasional inconsistency is tolerable and training infrastructure is unavailable. The expensive version is RuAE. It becomes attractive when evaluation is high-volume, high-stakes, and stable enough that training cost can be amortized.

The business value is not only higher accuracy. It is operational continuity. A learned-rule evaluator can preserve institutional scoring standards across products, departments, and model upgrades. That is especially important as companies increasingly replace one model with another while pretending the downstream evaluation system remains the same. Spoiler: it often does not.

Where the method should not be over-sold

The paper’s limitations are not cosmetic. They define the boundary of use.

First, the approach is strongest when tasks require complex analysis and reasoning. It offers limited improvement for tasks that are simple, short, or already well-handled by base models. Amazon review-star prediction is the clearest warning. Not every scoring task deserves MCTS plus RL. Sometimes a boring classifier is enough. Civilization may continue.

Second, the computational cost is real. MCTS rule search requires repeated LLM-based score prediction and reward calculation. RuAE training requires substantial GPU resources. For many companies, CoR will be the realistic entry point, while RL-trained evaluators will be reserved for mature, high-value evaluation pipelines.

Third, the action space is restricted. The system modifies rubrics mainly by making criteria stricter or more lenient. That makes search feasible, but it may miss richer forms of rubric transformation, such as splitting one criterion into two, merging overlapping criteria, or introducing conditional rules.

Fourth, the method assumes that unified rules are desirable. That is not always true. Some evaluation tasks benefit from plural standards, stakeholder disagreement, or context-specific judgment. Hiring decisions, creative assessment, political content analysis, and ethical trade-off evaluation can all involve legitimate disagreement. Forcing a single rubric onto such tasks may produce false consistency. Very tidy. Very dangerous.

Finally, learned rules inherit the values and blind spots of labeled data. If historical labels are biased, noisy, or strategically distorted, rule distillation will not magically discover justice. It will discover patterns. The difference remains annoying and important.

The evaluator needs law, not just eloquence

The easy story is that LLMs will become better judges because models will become larger and reasoning traces will become longer. The paper offers a better story: evaluators improve when their standards are learned, their rules are explicit, and their execution is trained or constrained.

That is a useful shift. It moves AI evaluation away from personality and toward procedure. It says the judge should not simply sound wise; it should know which criteria matter, how those criteria map to scores, and when its score distribution has drifted away from human-labeled reality.

For businesses, the lesson is not to rush into RL-trained evaluator infrastructure tomorrow morning. The lesson is to stop treating LLM judgment as a magical primitive. A judge without a stable rubric is not an evaluator. It is an opinion generator with API access.

Learned-rule evaluators are still early. They are costly, bounded, and dependent on labeled data. But the direction is right: from improvised judgment to learned standards; from clever prompts to auditable rules; from bigger judges to better law.

That may be the real rule of attraction here. The model becomes useful as a judge only when it stops being attracted to its own improvisation.

Cognaptus: Automate the Present, Incubate the Future.