The Reward Model Has to Move Too: R2M Tracks the Policy During RLHF

TL;DR for operators

A reward model can remain technically unchanged while becoming operationally stale. As the policy shifts during RLHF, the evaluator increasingly scores outputs from a distribution different from the one that shaped its original preference fit. Real-Time Aligned Reward Model beyond Semantics proposes R2M, which lets the reward model incorporate current policy-internal representations and updates only a small fusion module plus the scoring head rather than retraining the full evaluator.¹

The strongest result is not simply that more information helps. It does not. Versions that feed policy feedback into an unadapted reward model perform worse than vanilla RL, while matched Gaussian-noise feedback yields only limited gains. The paper’s case is therefore about learned synchronization, not passive access to policy internals. In the tested dialogue and TL;DR settings, that lightweight adaptation improves both downstream RLHF results and held-out reward-model preference accuracy. For operators, the relevant design question becomes how little of the evaluator must change to keep pace with the policy.

A fixed evaluator can become stale while the policy keeps moving

A strong reward model looks like a stable reference point during optimization. The problem is that the object being judged does not remain stable. Policy updates change which responses are likely, how they are expressed, and which weaknesses in the evaluator can be exploited. A correlation that was harmless near the original policy distribution can become a target once optimization repeatedly searches for outputs that score well.

That creates three operational choices: keep the evaluator fixed, periodically retrain it, or give it a cheaper adaptation path that follows policy change. R2M is designed around the third option.

The paper first argues that the current policy contains signals the reward model cannot see from text alone. Its deeper internal representations show stronger similarity among responses with the same preference label than across preference labels, and similarity is negatively associated with absolute reward-score differences. Appendix analysis also reports higher within-category similarity for reward-hacking and non-hacking responses than across the two categories. These analyses are mechanism evidence, not proof that hidden states are universally reliable preference labels. Their role is narrower: they motivate using the policy’s changing internal state as input to the evaluator.

R2M learns how to use policy feedback instead of merely exposing it

R2M takes the policy model’s last-layer hidden-state sequence for each query-response pair and compresses that sequence into a form usable by the reward model’s scalar scoring process. The mechanism is sequence-to-token cross-attention: the reward model’s reward-token representation queries the policy hidden states, producing one aggregated representation in the reward model’s feature space.

The system then blends that policy-informed representation with the reward model’s original representation. A schedule starts with heavier reliance on the pretrained reward representation and gradually increases the contribution of policy feedback, while retaining a minimum weight on the original embedding.

The efficiency choice is equally important. The reward-model language-model backbone stays frozen. Only the cross-attention module and scoring head are updated. Policy hidden states are refreshed during the policy forward pass, avoiding an extra policy pass for reward-model adaptation.

The no-training ablation shows why this architecture should not be described as simply “giving the reward model more context.” On LLaMA3 with RLOO, vanilla RLOO reaches 28.4% length-controlled and 30.2% raw AlpacaEval win rates. R2M without adaptation falls to 24.4% and 27.4%. Replacing the policy signal with matched Gaussian noise reaches 25.4% and 26.4%. Full R2M reaches 34.5% and 38.2%.

The likely purpose of these tests is mechanistic isolation. They support the claim that learned fusion matters and that useful information exists in the policy signal beyond generic perturbation. They do not establish that every policy hidden state contains robust preference information.

GREBT protects the learning signal when group rewards collapse together

R2M also addresses a different failure mode: sampled responses within a group can receive nearly indistinguishable rewards. When scores bunch together, the relative learning signal weakens even if the overall reward model is not obviously wrong.

The paper combines two objectives. Bradley-Terry loss preserves pairwise ranking pressure using the highest- and lowest-reward responses as a pseudo preference pair. Group Reward Entropy, or GRE, standardizes rewards within the response group and minimizes entropy over their softmax-normalized scores, encouraging more discriminative separation. The combined objective is GREBT.

The ablation is supportive but bounded. In the LLaMA3/RLOO setting, full R2M scores 34.5% length-controlled and 38.2% raw win rate. Removing Bradley-Terry lowers those to 31.5% and 35.7%; removing GRE lowers them to 32.3% and 36.2%. The paper also proves, under its regularity assumptions and entropy-based definition of group degeneration, that increasing the GRE weight reduces that degeneration measure. This is evidence for the paper’s specific scoring problem, not a general theorem that sharper reward distributions always correspond to better human alignment.

The benchmark gains are consistent across the tested RL backbones

The main benchmark evidence is comparative: matched RL algorithms with and without R2M under the paper’s chosen model and task settings.

The reward-model accuracy result is especially relevant to the paper’s argument. It connects downstream policy gains to an evaluator that becomes better at preference discrimination after adaptation, rather than only to a change in RL dynamics.

The paper also reports lower time and memory cost for this lightweight update design than full reward-model updating. That supports R2M’s positioning as an efficiency-oriented alternative to full iterative retraining, although the package does not provide a broad hardware or scale-sensitivity study.

The operational decision is how much evaluator adaptation to buy

What the paper directly shows: within the tested dialogue and summarization pipelines, updating a small policy-feedback fusion path and reward head can outperform keeping the evaluator static, updating only a conventional reward head, or injecting policy feedback without learned adaptation.

Cognaptus inference: teams operating RLHF pipelines should treat reward-model drift as a synchronization problem. If the policy is moving materially during optimization, “freeze the evaluator” is not a neutral choice; it assumes the original reward representation remains adequate on the new policy distribution. R2M suggests a middle layer between static evaluation and expensive full retraining: adapt only the interface that translates current policy state into reward decisions.

This changes what operators should measure. Beyond final benchmark score, a validation plan should compare static reward modeling, lightweight adaptation, no-training feedback, and noise controls. Those controls help distinguish genuine policy-state information from the possibility that an added pathway merely changes optimization through perturbation.

The proposed multi-node design extends this logic to infrastructure. By placing a lightweight cross-attention copy on the policy node, the paper analytically reduces communication from $B \cdot S \cdot D_p$ to $2 \cdot B \cdot D_p$, removing sequence length from the transfer term. That is a systems hypothesis, not a measured distributed result.

Where the evidence stops

The empirical coverage is limited to Qwen2.5-3B, LLaMA3-8B, Pythia-2.8B, the Skywork dialogue reward model, and a task-specific TL;DR reward model. Evaluation covers dialogue and summarization, not code, multimodal systems, or high-stakes domains. TL;DR uses GPT-4 judging rather than newly collected human preference labels.

The theory is similarly conditional. R2M’s tighter reward-misalignment bound requires positive post-fusion alignment quality plus boundedness and Lipschitz assumptions. The GRE result depends on differentiability, a positive-definite Hessian at the minimizer, and the paper’s own entropy-based degeneration definition.

R2M is therefore best read as evidence that adaptive reward modeling can be lightweight and operationally plausible, not as a general solution to reward hacking. Its strongest contribution is more specific: when the policy keeps changing, the evaluator may need a controlled way to change with it.

Cognaptus: Automate the Present, Incubate the Future.