Rules of Attraction: How LLMs Learn to Judge Better Than We Do

Opening — Why this matters now

In the last year, AI evaluation quietly became the industry’s most fragile dependency. LLMs are now asked to judge everything—from student essays to political sentiment to the quality of each other’s outputs. Companies use them to score customer emails, assess compliance risks, and even grade internal documentation. The problem is obvious: we’re relying on systems that struggle to agree with themselves.

The paper Learned‑Rule‑Augmented Large Language Model Evaluators fileciteturn0file0 enters this chaos with an unfashionably simple proposal: stop asking LLMs to improvise evaluation criteria. Teach them rules. Real ones. And let them learn these rules automatically.

The result is a quietly radical reframing of how AI systems might evaluate future AI systems.

Background — The limits of free‑form judgment

Today’s LLM evaluators generally follow one of two philosophies:

1. Zero‑shot judgment — “Here is the text. Give me a score.”
2. CoT‑style evaluation — “Here is the text. Think step‑by‑step using these principles I wrote at 3 a.m. in a panic.”

Neither scales. Human‑written rubrics are costly, inconsistent, and model‑dependent. Worse, LLMs do not reliably follow them. As the paper shows, even when given the same task, Qwen‑7B generates wildly dispersed “evaluation principles” (Figure 1, page 2), making consistency accidental rather than engineered.

The core failure modes:

• mis‑1: Human rubrics rarely match the statistical patterns in real data.
• mis‑2: Even perfect rules don’t stop an LLM from drifting away during execution.

This is how you get evaluators that sound authoritative but cannot agree on what “good” means.

Analysis — A rule‑learning engine for evaluators

The authors propose a two‑stage system:

1. Distill interpretable scoring rules from data using MCTS
2. Teach models to follow these rules (prompting or RL)

The elegance is in the machinery.

1. Rule Distillation via MCTS

Instead of searching the enormous space of all possible reasoning paths, the system searches over rules—compact, human‑readable statements like “Word Choice: 1–2 = confusing language; 5–6 = precise, vivid language” (Table 8, pages 13–14).

Each candidate rule set is treated as a node in a Monte Carlo Tree Search. The reward? How closely the LLM’s scoring under these rules matches human‑labeled data.

MCTS splits the search into two phases:

• Stage 1: Add new sub‑rules (exploration)
• Stage 2: Refine rubrics—stricter or more lenient (exploitation)

Crucially, this method avoids token‑level explosion by operating at the conceptual level of scoring criteria.

2. Chain‑of‑Rule (CoR)

Once rules are learned, you prepend them to prompts.

It’s the lightweight option: a structural upgrade to prompting.

3. Rule‑Augmented Evaluator (RuAE)

The heavyweight option is reinforcement learning. Using Group Relative Policy Optimization (GRPO), the model is trained to:

• match ground‑truth scores
• preserve ranking relationships
• follow the learned rules

This produces an evaluator that not only cites the rules—it reasons with them.

Findings — Where rules beat raw intelligence

Across four tasks—essay scoring (ASAP), citation relevance (Relish), rating prediction (Amazon), and summarization evaluation (SummEval)—the results are disarmingly clear.

1. CoR improves everything… except trivial short‑text tasks

Across multiple model families (DeepSeek, Qwen‑32B, Qwen‑7B), CoR consistently outperforms both vanilla scoring and CoT prompts.

• On SummEval: CoR beats even larger reasoning models.
• On Relish: CoR surpasses DeepSeek‑R1’s mAP by 52%.

Where it underperforms (Amazon), the task is too shallow to benefit from reasoning.

2. RuAE dominates long‑form, high‑complexity evaluations

Trained RuAEs outperform everything—including 671B‑parameter models—in:

• ASAP (essay scoring) — 20.3% above next best
• Relish (literature relevance) — +10% nDCG over CoR

The density plots (Figure 5, page 8) show why: RuAE’s predicted score distribution aligns almost perfectly with human ground truth.

3. Learned rules align strongly with human rules

On essays, the system rediscovers nearly the entire human rubric:

• Precision: 1.00
• Recall: 0.83
• Jaccard similarity: 0.83
• Hypergeometric significance: p = 0.024

A 66.7% improvement over random selection.

Illustration — Comparing Scoring Methods

Implications — Why businesses should care

1. Rule‑based evaluators are the future of AI assurance

Compliance teams today outsource far too much trust to opaque LLM judgment. A rule‑distilled evaluator—auditable, interpretable, data‑aligned—provides a foundation for:

• automated policy enforcement
• safe autonomous agents
• regulatory‑grade scoring frameworks

2. RL‑trained evaluators may form the backbone of future agent ecosystems

Agents need stable metrics to negotiate, verify, and coordinate. Free‑form reasoning is too noisy. Learned rules provide a shared language of evaluation.

3. The approach scales across domains without domain experts

Essay scoring, summarization, biomedical citation relevance, product sentiment—the system extracts rules that make sense in each domain.

A general evaluator moves closer to reality.

Conclusion — The new epistemology of AI scoring

LLMs have become prolific judges without ever learning the law. This paper’s contribution is giving them one—automatically, consistently, and aligned with real data.

Rule‑distilled evaluators are not just an upgrade to prompting. They are a blueprint for AI systems that reason with structure rather than vibes.

As evaluation becomes the backbone of governance, safety, and automation, this shift may matter more than any model size increase.

Cognaptus: Automate the Present, Incubate the Future.