Shape the Signal, Keep the Objective: MeRLa’s Bet on Reusable RLHF Rewards

TL;DR for operators

RLHF teams usually have two obvious levers: improve the reward model or improve the policy optimizer. Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback introduces a third.¹ MeRLa learns an additional task-aware reward signal across auxiliary tasks, freezes it, and adds it to the existing reward during subsequent policy optimization.

The reported gains are not confined to one optimizer. On AlpacaEval 2.0, adding MeRLa raises PPO from 78.6% to 85.2%, GRPO from 84.3% to 90.8%, and DAPO from 86.9% to 90.8%. It also remains additive when the base reward is already process-based or assembled from a rubric ensemble.

For organizations that repeatedly post-train models, the interesting possibility is architectural: reward informativeness may be separable from both reward-model construction and optimizer choice. A learned shaper could potentially be reused across runs while leaving existing reward infrastructure in place. The evidence remains bounded to LLaMA-3-8B-Instruct, and the paper’s policy-invariance argument is exact only when its potential-based conservation condition is satisfied exactly.

One reward signal has to guide very different tasks

A post-training pipeline may use the same reward infrastructure across mathematics, instruction following, open-ended response quality, and other prompt types. The desired direction can remain broadly correct while the useful distinctions differ substantially by task.

That creates an optimization problem. A reward that successfully ranks two completed answers may still provide a coarse learning signal for the many intermediate policy updates needed to separate stronger and weaker behavior. Changing the optimizer can improve how that signal is consumed, but it does not necessarily make the signal itself more discriminative.

MeRLa addresses that second problem. Chu’s method learns what the paper calls a shaping function: an extra signal designed to enrich the original reward rather than replace it. The deployed reward is

The distinction is operationally significant. Existing reward models remain part of the system. MeRLa inserts another learned layer between reward evaluation and policy optimization.

The shaper is learned once, then frozen before RLHF

MeRLa separates its work into two phases.

During meta-learning, a two-layer MLP learns from 64 auxiliary tasks. Its objective combines three elements: distinguishing preferred from rejected responses, maintaining an informative reward distribution through entropy regularization, and constraining the learned signal toward a potential-based form. The shaping network reads prompt and response embeddings produced by a frozen reference model.

During deployment, the shaping network no longer learns. Its output is added to the base reward while the language-model policy is optimized using an RLHF algorithm such as PPO, GRPO, or DAPO.

This separation matters because an unconstrained auxiliary reward could simply introduce another objective. MeRLa instead tries to add local guidance while limiting changes to what the base reward ultimately prefers.

The paper uses potential-based reward shaping for that constraint. Under the exact potential-based condition, the extra reward can alter the learning path without altering the optimal policy induced by the original reward. MeRLa regularizes its learned shaper toward that structure rather than assuming that any informative auxiliary score is safe to add.

Improvements survive changes in the optimizer

The strongest evidence for treating shaping as a distinct optimization layer is the backbone comparison.

All values are AlpacaEval 2.0 length-controlled win rates reported over three runs.

The purpose of this test is compatibility rather than a second main thesis. MeRLa improves all three tested optimizers, so the reported effect is not dependent on GRPO alone.

The broader main comparison is also favorable. MeRLa with GRPO reports 90.8% on AlpacaEval 2.0, 9.14 on MT-Bench, 53.4% on MATH, and 81.2% on IFEval. Against DAPO, the strongest listed baseline, those differences are +3.9 points, +0.33, +5.6 points, and +3.9 points respectively; the paper reports $p<0.05$ for these gains.

Within this experiment, changing reward informativeness appears to add value beyond changing the optimizer consuming that reward.

Useful shaping also has to remain stable

A learned shaper introduces another failure mode: its inputs may move during training.

If the shaping network reads representations from the evolving policy, the meaning of those representations can change as policy parameters update. The reward component can therefore drift even when the underlying prompt and response do not.

The paper’s frozen-versus-evolving encoder experiment is best read as a mechanism test. With an evolving encoder, shaping-output variance over 1,000 steps is reported at 0.087. With frozen reference embeddings, it falls to 0.002. Final AlpacaEval performance rises from 87.1% to 90.8%.

This makes the frozen encoder more than an implementation convenience. In the proposed architecture, it provides a fixed coordinate system for a reward component that must remain meaningful while the policy itself changes.

The ablations reinforce the same point about structure. Removing task discrimination reduces reported performance by about 3.7%; removing entropy regularization by 2.6%; removing conservation by 3.2%; and removing task sampling by about 5.5%. These tests do not establish that the chosen formulation is unique, but they do show that the reported result is not attributable to merely attaching an extra MLP to the reward pipeline.

Stronger base rewards do not eliminate the gain

A natural alternative explanation is that MeRLa compensates for an unusually weak base reward. The paper tests that possibility by pairing the method with richer reward designs.

With a standard reward model, MeRLa raises AlpacaEval from 84.3% to 90.8%. With a process reward model, it raises performance from 87.1% to 91.3%. With a rubric ensemble, the increase is from 88.5% to 92.0%.

The incremental gain narrows as the base reward becomes stronger, but it does not disappear in the tested configurations. MATH shows the same direction: 44.5% to 53.4% with the standard reward, 50.2% to 55.8% with the process reward, and 51.7% to 56.4% with the rubric ensemble.

For an alignment team, that changes the architecture decision. A shaping layer need not require replacing process rewards, rubric systems, or an existing reward-model pipeline. The paper supports complementarity within its tested setup.

The business case depends on repeated post-training

MeRLa adds a meta-learning stage costing roughly two GPU-hours on eight A100 GPUs for 64 meta-tasks. For a single training run, that is simply another expense. For an organization repeatedly aligning related models or rerunning RLHF under changing data and optimizer configurations, the economics could be different because the learned shaper is frozen before deployment and its cost may be amortized.

Cognaptus’ inference is therefore narrower than “MeRLa lowers RLHF cost.” The more defensible proposition is that teams with repeated post-training workloads may have another reusable asset to manage: not only reward models and optimizer recipes, but a learned transformation of reward information.

The paper also reports reaching 90% of final reward within 150 steps, versus 250 for DAPO and more than 350 for PPO, alongside 41% lower reward variance. If those stability effects transfer to other model families, they could affect experiment throughput and failed-run rates. This study does not establish that transfer.

Policy preservation is approximate in the trained system

The most consequential boundary concerns the conservation argument.

Exact policy invariance follows when the shaping function satisfies the potential-based condition exactly. MeRLa only regularizes its learned shaper toward that condition. The final reported conservation loss is 0.0034, with an implied maximum policy-ranking change of about 0.001.

That is small in this experiment, but it is not zero. The paper explicitly leaves the theoretical consequences of residual deviation incompletely characterized. Its incentive-alignment analysis also assumes that the base reward model is not adversarially corrupted.

External validity is narrower still. All main experiments use LLaMA-3-8B-Instruct. The study therefore provides substantial comparative evidence for one post-training environment, not evidence that the same gains will survive different model sizes, architectures, reward distributions, or evaluators.

Reward design may be its own optimization layer

MeRLa’s most relevant contribution is not another entry in the optimizer comparison table. It separates three decisions that are often collapsed together: what the system ultimately rewards, how informative that reward is during learning, and how the policy optimizer consumes it.

Within the reported LLaMA-3-8B setting, improving the middle layer produces gains across multiple optimizers and remains additive to stronger reward models. The accompanying ablations suggest that the gain depends on making the signal task-aware while constraining incentive distortion and stabilizing the representations on which the shaper operates.

For teams with repeated RLHF workloads, that is enough to justify examining reward shaping as a reusable part of the post-training stack. It is not yet enough to assume that the same layer will transfer safely across model families—or that approximate conservation is equivalent to a guarantee.

Cognaptus: Automate the Present, Incubate the Future.