Your Reward Model Is Also a Voting Rule: What Social Choice Changes About RLHF

TL;DR for operators

When thousands of annotators disagree about two acceptable model responses, the pipeline still has to decide how that disagreement becomes model behavior. That decision is often hidden inside familiar technical choices: how comparisons are sampled, whether votes are collapsed into majority labels, which reward-model class is fitted, and how aggressively a policy is optimized against the resulting score.

AI Alignment From Social Choice Perspectives by Halpern and colleagues¹ reframes this process as social choice. Its sharpest result is conditional but revealing: for a fixed prompt, with complete population pairwise preferences and an unrestricted reward class, Bradley-Terry reward learning ranks responses exactly by their Borda scores—their average pairwise success against candidates drawn from the comparison distribution.

That does not mean deployed Bradley-Terry reward models are simply Borda voting systems. The equivalence depends on assumptions that production systems usually relax. The survey’s broader contribution is to show what can happen when those assumptions change: majority preprocessing can implement a different collective-choice rule, redundant response variants can alter outcomes, restricted reward classes can violate unanimity, and pairwise data may be insufficient for welfare objectives that depend on preference intensity.

For model-governance and feedback teams, the resulting audit question is wider than “Does the reward model predict labels accurately?” It is also: What aggregation rule does this pipeline induce, which properties survive generalization, and what information has been discarded before optimization begins?

Human disagreement needs an aggregation rule

A standard RLHF description can make the reward model sound like a measurement instrument. Annotators compare responses; the model learns which responses people prefer; policy optimization then pushes generation toward higher reward.

That account is incomplete when annotators disagree for substantive reasons.

The paper begins from the population preference object

the probability that a randomly sampled member of the target annotator population prefers response $y$ to $y’$ for prompt $x$. Once these probabilities conflict across people and alternatives, converting them into one scalar reward is not merely estimating a hidden consensus. It requires a rule for combining preferences.

The survey’s Section 3 makes that rule visible in an idealized setting. A response’s Borda score is

so responses rank highly when they perform broadly well against comparison candidates. Under complete population pairwise information and an unrestricted prompt-level reward class, the finite Bradley-Terry optimum orders responses exactly the same way.

This is main theoretical evidence, not a deployment experiment. Its role is diagnostic: a standard reward-learning objective can carry an implicit collective-choice mechanism even when nobody explicitly selected one.

Preprocessing can change the collective decision

Once aggregation is recognized as part of system design, several implementation choices stop looking neutral.

The clearest example is majority-vote preprocessing. If the pipeline replaces soft population preference probabilities with binary majority labels before fitting the reward model, the survey reports that the induced rule becomes Copeland-style rather than Borda-style. Borda rewards broad average pairwise support; Copeland ranks alternatives according to how many head-to-head contests they win.

The annotation data can therefore describe the same underlying population while a preprocessing step changes the rule used to convert disagreement into ranking.

Candidate representation creates another dependency. Standard Bradley-Terry learning can be sensitive to approximate clones: semantically redundant responses occupying a dense region of the candidate space can change learned rewards and downstream probability mass. The reviewed uniqueness-weighted approach can provide approximate-clone robustness, but only relative to a specified embedding, metric, reference response space, and regularity assumptions on preferences.

For an organization running preference collection, this turns response generation into a governance-relevant input. If one semantic position is represented by many paraphrases and another by few, the team should test whether duplication itself changes the learned outcome.

Generalization can break unanimity

The Borda characterization is easiest to misread here. It applies when each candidate can effectively receive an unrestricted reward. Production reward models instead share parameters across responses and prompts so that they can generalize beyond observed comparisons.

That coupling can remove properties available in the unrestricted case.

The survey reviews a formal result in which a loss-minimizing linear reward model ranks one response below another even though every annotator prefers the first. The mechanism is not disagreement among annotators. Shared parameters force the model to balance margins across multiple comparisons, allowing optimization elsewhere in the hypothesis class to overturn a unanimous pairwise ordering.

For reward-model audits, aggregate preference-prediction accuracy therefore does not exhaust the relevant checks. A team responsible for alignment quality can construct unanimity tests: when all observed or modeled population preferences agree on a pair, does the fitted model preserve that ordering? Failure would identify a consequence of model restriction that ordinary held-out accuracy could conceal.

This is a design principle inferred from the paper’s formal synthesis, not evidence about how frequently unanimity violations occur in deployed systems.

Pairwise labels may not contain the welfare information you need

A more fundamental limitation appears when the organization’s objective depends on how strongly different people value outcomes.

Anonymous pairwise probabilities can be compatible with multiple underlying cardinal reward profiles when choices arise through nonlinear response links. Two populations can therefore generate the same observable pairwise preference probabilities while implying different utilitarian welfare rankings.

The survey analyzes this information loss through utilitarian distortion: the worst-case ratio between the welfare of the best policy and the welfare achieved by the selected policy across latent reward profiles compatible with the observed feedback.

This separates two questions that are easily conflated. A reward model may fit pairwise preferences well, yet the elicitation process may never have captured enough information to identify the welfare quantity the organization wants to optimize.

The reviewed remedy is not simply a better predictor. Richer elicitation—such as additional comparisons or larger choice sets—can reveal more welfare-relevant information. For products where preference intensity affects an explicit welfare or fairness objective, annotation design may therefore need to change before model architecture does.

One scalar reward is not the only architecture

The survey also reviews methods that avoid compressing all pairwise structure into a scalar reward before policy optimization.

Maximal lotteries retain a distribution over alternatives rather than forcing a deterministic winner when collective preferences are cyclic. Nash Learning from Human Feedback extends the idea to policies through a maximin objective:

The policy is chosen to maximize its worst-case pairwise preference probability against competing policies.

Under the reviewed normalized Bradley-Terry welfare model, maximal lotteries attain asymptotically minimax-optimal worst-case utilitarian distortion among rules operating on the same anonymous pairwise information. That is a specific theoretical guarantee under a specific preference model, not a general proof that Nash-style alignment dominates scalar reward learning.

Still, it changes the architecture available to teams working with persistent pluralism. Consensus outputs, multiple policies, user-matched policies, and mixtures can be treated as design options rather than assuming that every population must ultimately be represented by one global reward.

What teams can audit now

The paper supports a broader alignment audit across four stages:

Cognaptus infers that teams with heterogeneous user populations should document these choices alongside conventional reward-model metrics. A persistent disagreement rate is not, by itself, evidence of annotation failure. It may indicate that the product needs an explicit decision about aggregation, segmentation, richer elicitation, or policy plurality.

The boundary is mathematical, not empirical

This paper is a narrative survey built largely from formal characterizations, theorem results, derivations, and counterexamples. Its strongest conclusions hold within stated assumptions: complete pairwise information for some results, unrestricted reward classes for the Borda equivalence, specified response geometry for clone robustness, and normalized latent rewards plus Bradley-Terry-style models for several welfare guarantees.

It does not estimate how often clone sensitivity, unanimity violations, welfare distortion, or preference collapse materially affect deployed systems. Nor do the proposed alternatives eliminate normative choices. Clone weighting requires a notion of semantic similarity; welfare analysis requires deciding what welfare means; personalization can preserve heterogeneous preferences while also narrowing exposure to competing viewpoints.

The resulting lesson is narrower and more actionable than a claim that RLHF is fundamentally broken. Human-feedback alignment already contains collective-choice decisions. The engineering risk is allowing those decisions to remain implicit while evaluating only whether the resulting predictor fits its labels.

For systems serving genuinely heterogeneous users, the relevant governance object is therefore not just the reward model. It is the full path from who is asked, through how disagreement is represented, to which collective objective policy optimization ultimately amplifies.

Cognaptus: Automate the Present, Incubate the Future.