TL;DR for operators

Different answers to the same question across languages are not automatically evidence of cultural adaptation. They may indicate translation problems, degraded generation, or genuine changes in the model’s expressed position. A multilingual benchmark finds that one language with substantial apparent divergence falls below its factual-question noise baseline, while another retains a measurable residual difference.

The operational implication is to diagnose response quality before interpreting perspective differences. English-language performance is insufficient: models can present both sides of a controversy in English but produce more directional answers elsewhere. Ordinary contextual changes also alter positions, even without adversarial prompting. Organizations deploying multilingual assistants should evaluate response validity, argument coverage, stance stability, factual-control divergence, and prompt framing separately. None of these measurements alone establishes cultural appropriateness or production readiness.

A different answer does not necessarily mean a different perspective

A company deploys a public-information assistant across several languages. Asked about a disputed policy in English, the assistant explains the arguments for and against it. Asked an equivalent question in another language, it recommends one position.

The difference creates an immediate operational question: has the model adapted to a different audience, misunderstood the translated question, or failed to generate a competent response?

Research by Wiem Ben Rim, Yao Lu, Pontus Stenetorp, and Emine Yilmaz1 shows why these explanations cannot be treated as interchangeable.

In their DebateQA evaluation, Amharic responses showed a raw cross-language divergence score of 0.2483 relative to English. Yet the corresponding divergence on factual questions, where differing cultural perspectives should not explain the answers, was higher at 0.2724. Swahili, by comparison, retained a positive adjusted divergence of 0.0432, with a 95% confidence interval of [0.0293, 0.0567].

The larger apparent difference therefore did not provide the stronger evidence of a perspective shift. For someone deciding whether to approve a language-specific release, raw answer differences are observations requiring diagnosis, not explanations in themselves.

ROOTS measures argument differences while accounting for linguistic noise

The study introduces PerspectivesQA, a dataset containing 20 globally contested questions, 1,280 curated human arguments, and question translations across 18 languages. Its evaluation framework, ROOTS, examines two dimensions: differences between language-conditioned responses and sensitivity to question framing.

The principal cross-lingual experiments use nine languages, including English, across eight models from the Gemma 3, Qwen3, and Claude Haiku families. The researchers also use DebateQA for broader controversial-question evaluation and SimpleQA as a factual control.

A model’s conclusion is only part of its answer. Two responses can express similar final positions while selecting different arguments, omitting qualifications, or presenting opposing evidence with different emphasis.

ROOTS addresses this by extracting individual arguments from responses and representing them as multilingual embeddings. It applies Maximum Mean Discrepancy (MMD) to compare the distributions of those arguments with an English reference. A translation-based offset correction reduces systematic differences caused by language representation, while a separate back-translation approach provides another measure of divergence.

These two measures correlate across language means: 0.817 on DebateQA and 0.890 on PerspectivesQA. This comparison supports some consistency between measurement approaches, although their agreement does not independently validate cultural interpretation.

The framework also classifies responses as supporting a position, opposing it, presenting both sides, or remaining neutral. Factual-control comparisons then help assess how much apparent divergence might arise from linguistic or capability-related degradation.

The result is a more informative evaluation procedure than comparing final answers alone. It remains an imperfect decomposition: factual-question degradation is not necessarily equivalent to degradation when reasoning about contested issues.

The factual-control test changes how divergence should be interpreted

The paper’s factual-control comparison illustrates the difference between detecting response variation and identifying an excess beyond estimated linguistic noise.

Language Raw divergence SimpleQA baseline Adjusted difference
Amharic 0.2483 0.2724 −0.0240
Swahili 0.1570 0.1138 +0.0432
Kazakh 0.1658 0.1555 +0.0103

Source: Appendix A.3.2, Table 8. Values concern DebateQA and back-translated divergence relative to English.

Amharic’s measured divergence does not exceed its factual-control baseline. Swahili’s positive residual has a bootstrap interval excluding zero. Kazakh’s interval, [−0.0018, 0.0236], crosses zero and is therefore inconclusive under this calibration.

This distinction matters because lower-resource languages also exhibit substantial response-quality failures. Almost half of the Swahili and Amharic responses on PerspectivesQA were described as degenerate. The analysis filters irrelevant, refused, and degenerate responses, leaving a selected subset for subsequent comparisons.

Consequently, a high divergence score could coincide with an inability to generate usable answers. Conversely, a positive adjusted difference indicates divergence beyond the chosen factual baseline, not independently verified cultural adaptation.

For multilingual quality assurance, generation failures should therefore be recorded separately from substantive disagreements. Otherwise, an operator may classify a language-support failure as a localization feature.

English-language balance does not guarantee multilingual consistency

On PerspectivesQA, the primary automated evaluator classified 83.8% of admissible English responses as presenting both sides of a controversy. The corresponding aggregate on DebateQA was 49.2%, showing that even English response composition depends substantially on the question set.

Within PerspectivesQA, both-sided responses were less frequent in several other languages. Amharic and Kazakh were each approximately 32%, compared with roughly 69%–84% among English, Brazilian Portuguese, and Chinese.

These differences establish language-conditioned variation in response patterns, not the values of the populations speaking those languages. Some directional answers may reflect substantive differences; others may reflect comprehension failures or automated labeling errors.

Model size is associated with greater consistency. Across seven evaluated open-weight checkpoints, correlations between log parameter count and adjusted divergence range from −0.9341 to −0.9745 across the three datasets. Larger checkpoints generally diverge less, although lower-resource differences persist.

This is a checkpoint-level association, not a universal scaling law. Greater consistency also does not prove that the expressed viewpoints are culturally representative. A model can consistently reproduce the same dominant framing in every language.

Ordinary contextual changes can polarize an initially balanced answer

Language is not the only source of instability.

The researchers also test five minimally modified English prompts per question: three neutral contextual variants and two adversarial variants. Here, polarization means that a response initially classified as presenting both sides becomes directional after the framing changes.

On DebateQA, neutral contextual modifications polarize 21.4% of initially both-sided responses. Adversarial modifications polarize 27.4%.

The adversarial excess is 6.0 percentage points, with a paired-bootstrap 95% interval of [4.7, 7.3] percentage points.

Adversarial premises increase the risk, but ordinary contextual modifications already produce substantial polarization. A deployment test restricted to deliberately hostile prompts would miss this source of variation.

For an organization using configurable assistant templates, this suggests testing routine changes to surrounding context, not merely explicit attacks. The result does not show that every such change is inappropriate: it shows that the underlying question can remain the same while the expressed position changes.

The reported framing estimates come from English-language experiments. Their magnitude cannot automatically be transferred to other languages.

Multilingual release checks need separate failure categories

Cognaptus inference: The study motivates a layered evaluation workflow for organizations operating multilingual customer assistants, research systems, or public-information services.

Evaluation stage Operational purpose Decision informed
Translation and validity Check question equivalence and detect irrelevant, repeated, or unusable output Whether a language implementation is functioning
Argument coverage Compare substantive claims against a curated reference set Whether relevant viewpoints are being omitted
Divergence calibration Compare contested-question divergence with a factual-control baseline Whether differences warrant substantive investigation
Stance comparison Track balanced, neutral, and directional classifications Whether expressed positions change across languages
Framing robustness Test ordinary contextual variants and adversarial premises Whether prompt-template changes require further testing
Evaluator validation Compare judge outputs and inspect uncertain classifications How much confidence to place in reported differences

The sequence matters. If a model repeatedly produces unusable Swahili answers, calculating a stance-shift rate on the remaining valid outputs cannot substitute for diagnosing the generation failure.

Similarly, a change in the automated stance label should trigger examination of the arguments and response quality before a release owner concludes that the assistant has violated a policy.

Model procurement decisions should account for these language-specific failure modes alongside model size and English-language benchmark performance. This is a proposed operational application of the findings, not a deployment procedure validated by the paper.

Automated judges and cultural interpretation remain unresolved

The study’s evaluator comparison makes another boundary visible. The primary and secondary LLM judges agree on only 58.0% of individual DebateQA stance classifications and 66.2% of PerspectivesQA classifications. Yet their language-level rankings of both-sided response frequency correlate at Spearman values of 0.817 and 0.783, respectively.

Aggregate trends can therefore be more stable than individual labels. That is relevant to language-level monitoring, but it limits the reliability of treating a single automated classification as definitive evidence of a policy inconsistency.

Several other boundaries constrain deployment interpretation. PerspectivesQA covers only 20 contested questions, and its curated reference arguments originate from English-language debate platforms. Nine languages are used in the principal experiments despite translations being available in 18. The factual-control adjustment relies on an assumption about the comparability of degradation across factual and subjective questions.

Most importantly, the paper does not establish that language differences arise from particular alignment-training mechanisms, reproduce population opinion distributions, or cause measurable downstream harms. It supplies no empirically validated release thresholds.

Diagnose the failure before approving the explanation

The study changes the evaluation question. Rather than asking only whether translated prompts produce different answers, an operator needs to determine whether the model understood the question, generated usable arguments, exceeded its language-specific noise baseline, and maintained a defensible stance under ordinary contextual changes.

That approach is valuable for model selection, localization testing, and release governance. It does not guarantee cultural representativeness or eliminate subjective judgment.

A different answer is a signal worth investigating. The evidence needed to call it a different perspective is considerably stronger.


Cognaptus: Automate the Present, Incubate the Future.


  1. Wiem Ben Rim and Yao Lu and Pontus Stenetorp and Emine Yilmaz (2026). A Russian, a Brazilian, and a German Walk into a Bar, Ask the Same Question– And Get Three Different Answers: Profiling LLM Perspective Diversity on Contested Topics. arXiv:2610.34278. https://arxiv.org/abs/2610.34278 ↩︎