Value Collision Course: When LLM Alignment Plays Favorites

A support chatbot does not wake up one morning with a worldview. It gets one, slowly, through the dull machinery of product decisions: who labels the data, how many options they can choose from, whether disagreement is kept or ironed flat, and which optimization method gets the privilege of turning messy human judgement into model behaviour.

That is the inconvenient point in Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior.¹ The paper is not saying, “people disagree.” Congratulations, society has noticed. It is saying something more operationally useful: disagreement enters the alignment pipeline through specific technical gates, and those gates decide which values survive training.

For businesses deploying LLMs, that makes alignment less like buying a safety feature and more like designing a governance process. The model may appear neutral at the output layer, but neutrality is often just a tidy user interface over a rather opinionated annotation pipeline.

Alignment Is a Compression Pipeline, Not a Moral Destination

The ordinary alignment story is deceptively clean. Collect human feedback. Convert it into preferences. Train the model. Evaluate whether the model is safer, more helpful, less toxic, or more emotionally aware. Ship, announce “responsible AI,” and try not to read the audit logs too closely.

The paper breaks that story at the point where it matters: the conversion from plural human judgement into a single training signal.

Human feedback is not born as one neat label. It arrives as ratings from people with different backgrounds, assumptions, sensitivities, and thresholds for harm. The pipeline then compresses this variation. Sometimes it compresses by majority vote. Sometimes by averaging. Sometimes by discarding non-consensus cases. Sometimes by reducing a five-point judgement into a binary preference. Each move looks technically reasonable. Each one also decides what kind of human disagreement the model is allowed to learn from.

That is the mechanism-first reading of the paper. The important object is not merely the aligned model. It is the sequence of decisions that produces the model:

This is why the paper is useful. It turns pluralistic alignment from a seminar-room principle into an engineering surface.

The Paper Tests Where Values Get Flattened

The authors build a bilingual English-German alignment pipeline focused on gender-related prompt-response pairs. They start from red-teaming, gender-bias, and alignment datasets, generate responses using Wizard-Vicuna-7B-Uncensored-GPTQ, translate the pairs into German, and ask participants in the United States and Germany to rate model outputs.

The human-feedback dataset contains 1,095 participants and 27,375 ratings across five dimensions: toxicity, emotional awareness, sensitivity and openness, helpfulness, and stereotypical gender bias. Each participant rates five prompt-response pairs. The five-point Likert scale lets raters express intensity rather than being forced into a binary “acceptable/unacceptable” decision.

The study then uses the ratings in four fine-tuning experiments across seven open model architectures ranging from 1B to 14B parameters. The experiments focus mainly on toxicity and emotional awareness, two dimensions that capture different parts of alignment: avoiding harm and responding with social understanding.

The experiments have different evidentiary roles:

The paper’s core contribution is not a new morality. It is a controlled demonstration that annotation design and optimization design materially alter model behaviour.

Demographics Change the Signal Before Training Begins

Before fine-tuning, the authors examine how participants rate identical prompt-response pairs. This matters because any alignment process that treats ratings as interchangeable assumes that “human feedback” is one thing. The data says otherwise.

The paper finds disagreement in 85.3% of prompt-response cases evaluated by multiple participants, ranging from 84.5% for sensitivity to 86.2% for helpfulness. This is not a marginal annotation-cleaning problem. It is the normal state of value judgement.

The regression results show systematic demographic effects. Male participants rate responses as 18% less toxic and 20.9% less stereotypically biased than female participants. Conservative participants rate responses 27.9% more sensitive and 27% more emotionally aware than liberal participants. Black or African American participants rate responses 58% more sensitive and 44% more emotionally aware than White participants. Participants aged 51–60 rate responses 40.6% less helpful than younger participants.

These findings do not mean demographic identity mechanically determines judgement. The paper is careful enough to report low-to-moderate agreement even within groups. For example, Krippendorff’s alpha values for toxicity annotations remain far from unanimity across the subgroup comparisons. That detail is important. It prevents the lazy reading where “women think X” or “conservatives think Y” becomes the headline. The actual point is subtler: demographic composition affects the statistical signal available to the alignment pipeline, even though no group is internally monolithic.

Then the authors test whether these rating differences survive fine-tuning. They do.

Models trained on liberal feedback produce higher emotional-awareness scores than models trained on conservative feedback, with a pooled effect of 0.049 and $p = 0.010$. Models trained on White feedback similarly produce higher emotional-awareness scores than models trained on Black feedback, with a pooled effect of 0.046 and $p = 0.001$. Models trained on female feedback show lower toxicity than models trained on male feedback, with a pooled effect of -0.035 and $p = 0.002$.

The practical interpretation is not that product teams should pick a favourite demographic and call it governance. Please do not give that slide to Legal. The point is that model behaviour shifts when the feedback source shifts. Rater composition is therefore not a procurement detail. It is part of the model specification.

A useful additional finding: these effects are dimension-specific. Female toxicity feedback does not significantly alter emotional awareness, and liberal or White emotional-awareness feedback does not significantly alter toxicity. That suggests the model is not simply becoming globally “more aligned” or “less aligned.” It is learning particular behavioural preferences from particular feedback dimensions.

For business use, this is good news and bad news. Good, because targeted alignment may be more controllable than vague value-washing. Bad, because it means companies cannot hide behind general safety claims when their training pipeline has very specific behavioural consequences.

Disagreement Is Load-Bearing Data

The most commercially useful result may be the least glamorous one: preserving disagreement works better than pretending consensus exists.

The paper compares five ways of handling inter-annotator disagreement for toxicity training: preserving all ratings, averaging ratings, majority vote, random selection, and full consensus. All methods reduce toxicity relative to the unfine-tuned control, but not equally.

Preserving all ratings produces the strongest toxicity reduction, with an effect size of -0.242. Averaging is close behind at -0.229. Majority vote drops to -0.158. Random selection reaches -0.146. Full consensus barely moves the needle at -0.039.

The result is not just philosophically tidy. It is operationally sharp. Preserving all ratings is about 53% more effective than majority vote and nearly six times more effective than full consensus in reducing toxicity.

This reverses a common instinct in AI operations. Many teams treat disagreement as an annotation-quality problem: if raters disagree, the label must be noisy; if the label is noisy, aggregate it; if aggregation is hard, keep only the cases where everyone agrees. Clean dataset, clean conscience.

The paper shows why that instinct can be expensive. In subjective safety tasks, disagreement can contain the very signal the model needs. A response that some raters perceive as toxic and others do not may be exactly the case where the model needs a more nuanced behavioural update. Filtering it out because it is uncomfortable is not rigour. It is sweeping the useful dirt under the clean-data carpet.

There is a business analogue here. Customer complaints are rarely evenly distributed. The first group to notice a harm pattern may be small, sensitive, or demographically specific. A majority-vote alignment pipeline can convert that early-warning signal into “not statistically dominant enough to matter.” The model then learns the majority’s comfort zone and calls it safety.

That is how systems play favourites without anyone explicitly asking them to.

Binary Feedback Is Cheap Until It Throws Away the Gradient

The second technical lever is rating granularity. The authors compare toxicity training using five-point, three-point, and binary versions of the feedback.

All three reduce toxicity relative to the control. But the five-point scale performs best, with an effect size of -0.242. The three-point scale follows at -0.225. The binary scale reaches -0.198. The five-point scale outperforms binary feedback and is reported as 22% more effective at reducing toxicity.

This is one of those findings product teams should read before simplifying annotation tasks in the name of cost. Binary feedback is attractive because it is fast, clean, and easy to model. Accept or reject. Safe or unsafe. Good or bad. Humans love binaries when dashboards are involved.

The problem is that subjective judgements often have intensity. “Mildly inappropriate” and “severely toxic” are not the same training signal. “Somewhat emotionally aware” and “deeply responsive to the user’s distress” are not equivalent. When a pipeline collapses those distinctions, it may save annotation time while losing the gradient that makes the model update more effectively.

This does not mean five-point scales are universally superior. The paper’s scale comparison is constructed by converting the original five-point ratings into coarser versions, and the authors note that directly collecting ratings at different granularities would be a cleaner future test. Still, within this setup, the result is clear enough for governance: do not assume lower-friction feedback is free. Sometimes it is just cheaper because it has thrown away the expensive part: information.

Optimizer Choice Is Not a Fashion Contest

The fourth experiment compares Direct Preference Optimization with Group Relative Policy Optimization in a multi-value setup combining toxicity and emotional awareness. The result is a useful reminder that alignment methods should be benchmarked, not adopted by vibes.

DPO clearly outperforms GRPO in this tested setting. The DPO model trained on both toxicity and emotional awareness achieves a toxicity reduction of -0.159 and an emotional-awareness improvement of 0.084. GRPO produces smaller effects: -0.020 for toxicity and 0.029 for emotional awareness. The paper describes DPO’s effects as roughly eight times larger for toxicity and nearly three times larger for emotional awareness.

The authors also compare single-objective and multi-objective DPO. For toxicity reduction, DPO trained specifically on toxicity performs best at -0.243. The multi-objective DPO model is weaker for toxicity than the single-objective toxicity model. For emotional awareness, however, the DPO variants are statistically similar: DPO-EA reaches 0.083, DPO-Toxic reaches 0.068, and DPO-Toxic+EA reaches 0.084, with no significant pairwise differences.

The mechanism is not mysterious. Multi-objective optimization sounds elegant, but combining objectives can dilute the signal for a specific behaviour. If the business goal is to aggressively reduce toxicity in a defined domain, a focused training objective may be easier to interpret and more effective. If the goal is a balanced behavioural shift across several dimensions, the trade-off becomes more complex.

The paper does not prove DPO is always better than GRPO. It proves that in this data format, model set, evaluation setup, and two-objective alignment task, DPO is the stronger choice. That distinction matters. Method branding is not methodology. “We used the latest optimizer” is not an audit result.

The Business Lesson Is Alignment Governance, Not Demographic Theatre

The wrong business takeaway is: “Let’s align one model to every customer segment and declare pluralism solved.” That way lies product chaos, compliance confusion, and probably a meeting with someone whose title includes “risk.”

The better takeaway is that alignment needs governance over the feedback supply chain. A company deploying LLMs in customer support, education, healthcare-adjacent triage, HR, finance, or public-facing moderation should know which values its model has been trained to prioritize and which were silently weakened by preprocessing.

A practical governance pathway follows directly from the paper:

This is where the paper becomes useful to executives without becoming simplistic. The direct result is empirical: design choices change alignment outcomes. The Cognaptus inference is managerial: alignment should be audited like a pipeline, not described like a principle.

In practice, that means maintaining documentation for rater sourcing, demographic balance, rating definitions, aggregation policies, disagreement retention, objective design, optimizer choice, and evaluation method. It also means reviewing these choices periodically, because values and user populations shift. Alignment is not something a company “did” last quarter. It is something the system keeps doing, whether governed or not.

Where the Evidence Stops

The study has boundaries, and they matter.

The participant pool comes from the United States and Germany, recruited through Prolific. That gives the paper useful cross-country variation, but still within WEIRD-dominant contexts. The authors also note underrepresentation among conservatives, gender minorities, and older adults. This limits how far the demographic findings should be generalized.

The task domain is gender-related prompt-response pairs. That is a meaningful and safety-relevant domain, but it is not the whole world of alignment. Results might differ for medical advice, political persuasion, financial guidance, religious content, legal explanations, or multilingual public services outside English and German.

The evaluation uses GPT-4o-mini to score toxicity and emotional awareness. The authors validate the evaluator against two human experts on 50 responses and report 85% agreement. That helps, but does not remove all concerns about evaluator bias or cultural nuance. Automated evaluation is still part of the measurement pipeline, not a view from heaven.

The optimization comparison focuses on DPO and GRPO. It does not settle questions about PPO, Constitutional AI, reward-model variants, inference-time steering, or systems that preserve plural outputs instead of collapsing them into one behavioural policy.

The scale-granularity experiment converts original five-point ratings into three-point and binary versions. That is systematic, but not identical to asking people to provide ratings in those formats from the start. Anyone designing annotation workflows should treat this as strong evidence against casual binarization, not as the final word on scale design.

None of these limitations dissolves the central result. They define its proper use. The paper is not a universal theory of human values. It is a concrete demonstration that the alignment pipeline has knobs, and those knobs have consequences.

Pick Your Values Explicitly, Because the Pipeline Already Is

The central misconception this paper corrects is that disagreement is a defect to be averaged away. In many alignment tasks, disagreement is the evidence. It tells the model where human judgement is socially loaded, context-sensitive, or unevenly distributed across groups.

The second misconception is that inclusivity necessarily competes with safety. In this study, preserving more of the human signal improves toxicity reduction. Richer ratings outperform binary feedback. Disagreement-preserving methods beat majority vote. The inclusive choice is not merely nicer. It is technically stronger in the tested setting. How terribly inconvenient for anyone hoping ethics could stay in the appendix.

For business leaders, the article-sized lesson is simple: LLM alignment plays favourites when the feedback pipeline does. If a company does not know whose ratings dominate, how disagreement is handled, and which objectives its optimizer is privileging, then it has not built a neutral model. It has built an unexamined one.

And unexamined systems do not become fair by accident. They merely become confident.

Cognaptus: Automate the Present, Incubate the Future.