No Click Is Not a No: Turning Passive Feedback Into Reward Signals

TL;DR for operators

If your product already has thousands of clicks, copies, likes, shares, and non-actions, the tempting shortcut is to use those events as cheap preference labels. The problem is that the same no-action can mean a disliked response or a satisfactory response from a user who simply did not act. The event log records behavior, not preference directly.

ImplicitRM addresses that ambiguity by modeling two processes separately: whether the user preferred the output and how likely that user was to express the preference through an observable action. On PKU-SafeRLHF, the full method reaches an $R^2$ of 0.7872, versus 0.6848 when false-negative handling is removed and 0.6464 when adaptive action-propensity modeling is removed. The ablation therefore supports the need for both parts of the mechanism.

For operators, the opportunity is not simply to replace explicit preference labels with clicks. It is to treat interaction telemetry as partially observed preference data that may support ranking, data selection, or RLHF after the behavior-generating process is modeled. The evidence remains controlled, however: feedback is simulated, positive actions are assumed to reflect genuine positive preference, and the unbiasedness guarantee depends on correct latent-state estimates. Real production logs still need their own validation.

The same non-action can encode different preferences

Imagine two users receive equally satisfactory model responses. One copies the answer into a document. The other reads it, gets what they need, and closes the window.

A binary telemetry table records one positive event and one non-event. A naive reward model can easily learn that the first response was preferred and the second was not. The problem is that the behavioral difference may belong to the users rather than the responses.

The paper formalizes this distinction with a simple observation model:

Here, $r_i^{\ast}$ is the preference the system would like to know, $a_i$ indicates whether the user takes an observable action, and $r_i$ is the feedback the product actually records. Positive observed feedback therefore requires both a positive preference and an action.

That leaves three different explanations for no-feedback: the user disliked the response but acted, liked it but remained passive, or disliked it and remained passive. Collapsing those states into one negative class contaminates the training target.

This is the problem addressed by ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment.¹

ImplicitRM models preference and behavior as separate processes

ImplicitRM assigns probability to four latent groups: positive-active, negative-active, positive-passive, and negative-passive. The reward model is then trained using the estimated probability that each observation belongs to a positive or negative preference state rather than giving every no-feedback observation a hard negative label.

A second estimator models action propensity: how likely a user is to take the observable action conditional on the interaction. This matters because the probability that a non-action hides a positive preference need not be the same for every response or user.

The resulting preference objective weights each observation by posterior membership in the four groups. In simplified terms, probability mass assigned to positive-active and positive-passive states trains the positive side of the reward model; probability mass assigned to negative-active and negative-passive states trains the negative side.

The paper’s theoretical contribution is precise but narrower than the word “unbiased” can suggest. The preference objective equals the ideal preference objective in expectation when the estimated group probabilities equal the true posterior probabilities. The theorem therefore describes what happens under correct latent-state estimation. It does not establish that ordinary click logs automatically reveal unbiased preferences.

The ablation is stronger evidence than the headline comparison

The most informative experiment is Table 3 because it tests whether the two pieces of the proposed mechanism are doing distinct work.

On PKU-SafeRLHF, the naive objective produces an $R^2$ of 0.5535. A partial ImplicitRM variant without user-preference-bias handling reaches 0.6848. The variant without false-negative handling reaches 0.6464. With both mechanisms enabled, $R^2$ rises to 0.7872.

That experiment is an ablation rather than a second benchmark thesis. Its purpose is to isolate the contribution of the components. The result supports the paper’s mechanism: recovering hidden positive preferences among no-feedback observations is not sufficient by itself, and neither is adjusting for heterogeneous action behavior. The strongest result comes from modeling both.

The broader benchmark comparison points in the same direction. Across simulated implicit versions of HelpSteer, UltraFeedback, and PKU-SafeRLHF, ImplicitRM reports $R^2$ values of 0.3114, 0.5207, and 0.7872. Its corresponding RMSE values are 0.2919, 0.3169, and 0.2294, the lowest reported in the supplied table.

One detail is worth preserving: the table does not support a clean sweep on MAE. Some baselines report lower MAE on individual datasets. The defensible reading is that ImplicitRM leads the reported comparisons consistently on $R^2$ and RMSE, while its advantage depends on the evaluation metric for MAE.

The robustness tests ask whether the gain survives model and feedback changes

The backbone experiment is best read as a robustness test. On PKU-SafeRLHF, the method improves $R^2$ over the naive objective across Qwen3-8B, 14B, and 32B and LLaMA2-7B and 13B. Relative improvements range from 31.5% to 63.2%.

That reduces the chance that the result is merely a favorable interaction with one backbone. It does not establish architecture independence in a broad sense; five tested models remain a bounded sample.

The action-propensity sweep tests a different sensitivity. As the share of positive samples that generate visible feedback rises from $\alpha=0.1$ to $0.9$, all methods improve and the gap narrows. That pattern is consistent with the proposed mechanism: when positive preferences are easier to observe directly, there is less hidden-state ambiguity to correct. ImplicitRM nevertheless remains ahead across the tested settings.

Better reward estimation survives into downstream safety training

The paper also asks whether the reward-model improvement matters after the reward model is inserted into an RLHF pipeline.

Using the learned rewards for GRPO, ImplicitRM produces the highest reported Safety Score for both tested policy models on HarmBench, StrongReject, and WildGuardMix. With Qwen2.5-Instruct-7B, the scores are 0.9258, 0.9710, and 0.8827 respectively, compared with 0.8381, 0.9007, and 0.7654 under the naive reward objective.

This is downstream comparative evidence, not evidence that the system is generally safer across deployment conditions. It shows that, inside this controlled pipeline and these three safety benchmarks, the improved reward objective survives the transition from reward prediction to policy optimization.

That connection matters because a reward model can look statistically better without necessarily changing policy behavior in a desirable way. Here, the paper provides evidence for both stages.

For product teams, the data contract changes

The business inference is narrower than “implicit feedback replaces annotation.”

A product with abundant behavioral telemetry may be able to extract preference information from data it already collects, but only if it stops treating the event log itself as the target. The target is latent user preference; clicks, copies, likes, and non-actions are observations generated by both preference and user behavior.

That distinction changes several operational decisions. A ranking system could use a propensity-aware preference estimator rather than optimizing raw engagement. A data-selection pipeline could prioritize responses that appear genuinely preferred rather than merely action-inducing. An RLHF pipeline could supplement explicit preference collection with historical product interactions, provided the latent-state model is validated for the product’s own users and actions.

The ROI claim is therefore conditional. The paper suggests a path toward reducing the marginal amount of explicit labeling required. It does not yet quantify how many labels can be removed from a production system without degrading reward quality.

Production logs introduce problems the experiment does not test

The largest boundary is external validity. The reward-model experiments simulate implicit feedback from datasets that retain explicit preference labels, allowing the authors to compare predictions against an oracle. Naturally occurring click or copy logs may contain additional structure that this setup does not reproduce.

Observed positive actions are also assumed to indicate positive preference. A misclick, accidental copy, curiosity click, or strategically induced engagement violates that assumption. The current formulation does not model noisy positive actions.

Finally, the theorem’s unbiasedness condition depends on correct posterior probabilities for the latent groups. Production teams do not observe those true posteriors directly. Action propensities may also shift across user segments, interface designs, tasks, and product releases.

Those are not peripheral caveats. They determine whether the method can convert real telemetry into a better reward signal rather than a more elaborate behavioral proxy.

The next test belongs on real interaction logs

ImplicitRM provides a more credible way to think about passive feedback because it separates the preference a product wants to learn from the behavior it happens to observe. The ablations support that decomposition, the benchmark results show gains across several datasets and backbones, and the downstream experiments show that the difference can propagate into policy training.

The unresolved question is whether the latent-state estimates remain reliable when the feedback is generated by actual users rather than simulated from explicit labels.

For operators already sitting on large interaction datasets, that makes the next decision reasonably concrete: do not ask whether clicks are cheap preference labels. Ask whether your system can estimate preference and action propensity separately well enough that the resulting reward model beats one trained from the raw events. That is the production test this paper makes worth running.

Cognaptus: Automate the Present, Incubate the Future.