When the Reward Is Right but the Incentive Is Wrong

TL;DR for operators

A preference score can rank responses correctly and still be the wrong signal to feed directly into an alignment system. Wang et al. show why: when deployment deliberately keeps the aligned model close to its base behavior, the base model’s own response probabilities continue to influence what gets generated. Their Stackelberg Reward Shaping framework therefore changes the reward landscape rather than simply increasing reward strength.¹

Under the paper’s assumptions, the optimal bounded reward is prompt-specific: responses above a utility threshold receive the maximum reward, while those below it are suppressed. The threshold can be estimated from samples of the base model, and a smoother sigmoid version makes the rule less brittle in practice.

For teams operating preference-guided assistants, this changes the design question. Instead of asking only how strongly decoding should follow a reward model, ask whether the score should first be transformed to compensate for the base model’s existing bias. The paper’s experiments support that approach across its tested inference-time settings, but they do not establish a general recipe for train-time alignment or for reward signals that encode unsafe preferences.

A correct preference score does not erase the base model

Suppose a reward model reliably gives better responses higher scores. It is tempting to treat deployment as a tuning problem: increase reward strength until the assistant follows those scores closely enough.

That logic misses what happens when the aligned policy is deliberately constrained to stay near the original model. In the paper’s KL-regularized formulation, alignment does not replace the base distribution with a reward-defined distribution. It tilts the base distribution:

The reward changes relative probabilities, but $\rho_{\mathrm{base}}$ remains inside the deployed policy. If the base model already places too little probability on responses the user prefers, merely reporting utility faithfully does not necessarily produce the highest achievable user utility.

Increasing the global reward strength $1/\beta$ can push harder against that bias. It also permits greater departure from the base model, which can amplify reward-model errors or reward hacking. The paper therefore treats the operational reward as something to design, not just something to estimate.

The optimal bounded signal depends on the prompt

The paper formalizes that design problem as a leader-follower game. The reward provider chooses a bounded reward signal; the model then responds by optimizing that signal while remaining KL-regularized toward its base policy.

The resulting theoretical solution is unusually concrete. For each prompt, there is an optimal threshold $m^\ast(\mathbf{x})$. Responses whose underlying utility is above that threshold receive reward $B$; responses below it receive zero. The threshold itself is self-consistent: it equals expected utility under the policy induced by that shaped reward.

That structure matters because it is not equivalent to multiplying every reward by a larger constant. Two prompts can require different thresholds because the base model assigns different probability mass to their candidate responses.

It is also not simply “reward the single best answer.” Several responses can sit above the threshold. Which responses should receive the strongest incentive depends jointly on their utility and on the distribution the base model already supplies.

The reward bound adds another control. Under the paper’s inference-time assumptions, the induced policy satisfies

where $T$ bounds response length. This does not make bounded rewards sufficient for safe alignment, but it gives $B$ a specific governance role: the allowable reward range helps constrain how far reward-guided decoding can move from the base policy.

The theorem becomes usable through sampling and smoothing

A prompt-specific optimum would have limited deployment value if computing it required enumerating possible responses. The paper instead derives a helper function with a unique root and estimates that root from Monte Carlo samples generated by the base model.

In the main experiments, the method uses $M=10$ samples and solves the empirical root equation with bisection. Larger $M$ can improve estimation, but the paper reports diminishing reward gains relative to the additional inference cost. This is an implementation tradeoff rather than a second theoretical claim.

The exact threshold also creates a practical problem: tiny changes in utility near the cutoff can abruptly flip a response from zero reward to the maximum. The authors therefore introduce soft SRS:

The parameter $\alpha$ controls how sharply the reward transitions around the threshold. At finite values, the sigmoid sacrifices the discontinuous theoretical rule for a smoother operational control; as $\alpha$ increases, it approaches the hard threshold.

The main experiments support shaping rather than stronger scoring alone

The central empirical test integrates SRS into two existing inference-time alignment procedures: ARGS and Controlled Decoding. The experiments span HH-RLHF and Stanford SHP with Qwen3-8B and Llama3-8B-Instruct.

SRS records the highest average reward in all four principal evaluation settings while diversity and prompt-response coherence remain broadly comparable to the tested baselines.

The missing CD results are meaningful. Vanilla Controlled Decoding did not improve the Llama-based Eval-3 and Eval-4 settings before reward shaping, so the paper does not report SRS-CD there. SRS therefore improves both tested alignment mechanisms where the underlying mechanism itself is viable, but the evidence does not show that reward shaping rescues an ineffective decoding method.

Reward-model scores alone would be weak evidence because the shaped system is explicitly designed around rewards. The paper therefore adds GPT-4 head-to-head judgments on 300 sampled prompts. Across the reported comparisons, SRS obtains average win-tie rates of 66.83% against vanilla alignment, 69.6% against Minmax shaping, and 66.65% against Meanstd shaping. These judgments do not eliminate evaluator dependence, but they provide evidence that the gains are not confined to the optimized reward metric.

The additional tests probe robustness, not a broader thesis

The paper also perturbs several parts of the setup. These experiments are best read as sensitivity checks on the proposed mechanism rather than independent demonstrations of general alignment capability.

Serving cost also depends on integration. SRS-CD performs shaping offline and therefore adds no shaping overhead during inference. For ARGS, the reported threshold computation adds about $2.69\times10^{-4}$ seconds per token, approximately 0.03 seconds for a 128-token response, compared with roughly 14.05 seconds per ARGS prompt in the paper’s setup.

Treat the utility model and the deployment incentive as separate components

Cognaptus’ inference from these results is architectural. A preference model need not serve simultaneously as the best estimate of user utility and the exact control signal used during decoding.

That separation is relevant to teams operating personalized assistants or other preference-guided systems without retraining base-model weights. If a stable base model systematically underproduces a class of responses preferred by a target user, a deployment layer can in principle reshape incentives according to both preference scores and the base model’s existing distribution.

It also provides a more explicit quality-control surface. Reward bound $B$, shaping strength $\alpha$, reward strength $1/\beta$, and threshold-estimation budget $M$ become distinct controls with different effects. Collapsing all of them into “follow the reward more strongly” obscures those tradeoffs.

The business case is therefore not that SRS proves a universal alignment method. It is that reward transformation can be treated as its own system-design decision when a product already uses reward-guided decoding and wants to improve preference fit without modifying model weights.

The boundaries matter before deployment

The formal result belongs to a specific model: bounded rewards, a KL-regularized follower, and a reward provider able to choose an operational signal distinct from underlying utility. Moving outside those assumptions requires new analysis.

The empirical evidence is also concentrated on inference-time alignment. The authors leave train-time methods such as GRPO for future work, so the results do not establish that the same design will improve post-training.

Most importantly, SRS optimizes the preferences represented by the reward model. The paper assumes those preferences are benign and aligned with human values. If the reward model encodes malicious or otherwise undesirable objectives, better incentive design can make the system better at pursuing the wrong target.

Within those boundaries, the paper changes a useful deployment assumption. A reward model is not necessarily an instruction that should be followed as faithfully as possible. Under constrained alignment, it can be an estimate of what the user values, while the signal that most effectively induces those values may need to be designed separately.

Cognaptus: Automate the Present, Incubate the Future.