English-Like Is Not the Same as Correct: Measuring Multilingual Reasoning Quality

TL;DR for operators

A multilingual reasoning system may generate several plausible traces for the same request. The difficult part is deciding which trace to reward, trust, or return.

The evidence here argues against using one English-derived trace-quality score as that decision rule. Across ten languages, many reasoning features are associated with correctness in broadly similar directions, but their effect sizes vary substantially and sometimes reverse. English semantic and structural similarity are generally positive signals, so the paper does not show that English-like reasoning is harmful. It shows that English similarity is not consistently the strongest signal.

On the harder AIME benchmark, measures of whether reasoning steps actually contribute to the final answer—especially direct and indirect utility—along with result consolidation, consistently beat random candidate selection across the evaluated models. Direct-utility selection improves accuracy by up to 10% in the reported experiments.

For multilingual evaluation, reward design, and selective best-of-n routing, the practical move is therefore to compare several process signals rather than promote one English-centric proxy into a universal control rule. The boundary is substantial: the evidence covers mathematics, four reasoning models, and a selection procedure that generates and re-ranks 32 candidates.

A signal can transfer across languages without transferring cleanly

Suppose a multilingual system has several candidate reasoning traces and needs a scoring rule. A natural choice is to ask which candidate most resembles a strong English trace. If English reasoning is well represented in training and evaluation data, similarity looks like a reasonable proxy for quality.

The difficulty is calibration. A feature can be positively associated with correctness in aggregate while becoming weak, much stronger, or even negative in a particular language. That means transferring the direction of a signal is not the same as transferring its usefulness as a reward or ranking rule.

Ki, Duh, and Carpuat examine exactly this problem in What Makes Good Multilingual Reasoning? Disentangling Reasoning Traces with Measurable Features.¹ Across two mathematical-reasoning benchmarks, four open-weight reasoning models, and ten languages, they find that most trace features have broadly consistent directional relationships with correctness. The more consequential result is the heterogeneity underneath that average: effect magnitudes differ substantially by language, and some feature-language relationships reverse direction.

The paper treats these regression results correctly as associations, not causal effects. Increasing a feature mechanically would not necessarily increase accuracy.

Sixteen features turn reasoning traces into something operators can inspect

The study decomposes each reasoning trace into 16 measurable features across three dimensions.

The first dimension asks whether non-English reasoning aligns with English reasoning. It includes translation quality, semantic similarity, and what the paper calls structural similarity: whether two traces follow similar sequences of reasoning behaviors even when their wording differs.

The second asks whether individual reasoning steps contribute useful information. Validity measures whether a step is logically consistent with its dependencies. V-Information measures whether supplying the trace raises the model’s log-probability of the correct answer. Most operationally interesting are the dependency measures: some steps matter because they sit on a path that actually supports the final answer. The paper calls the proportion of such steps direct utility; indirect utility captures steps that support those directly useful steps.

The third dimension measures reasoning-flow behaviors such as computation, self-checking, problem setup, plan generation, result consolidation, and uncertainty management.

The broad pattern is more informative than any single coefficient:

One result sharpens the point about English similarity. On AIME, structural similarity has a larger estimated association with accuracy than semantic similarity for all four evaluated models. Even within “English-likeness,” the choice of what kind of resemblance to measure matters.

Harder problems make process utility more valuable

Association alone is insufficient for an inference system. A feature can correlate with correct answers without being useful for choosing among candidate traces.

The paper therefore runs a controlled best-of-32 selection experiment. For each query, it generates eight candidates at each of four temperatures, then changes only the feature used to select among those candidates. Selected pass@1 is compared with random selection using paired-bootstrap tests.

On the easier MGSM-Rev2 benchmark, most feature-based selection produces limited gains. That fits the earlier analysis: when feature effects are weak, a ranking policy has little leverage.

AIME produces a different pattern. Direct utility, indirect utility, and result consolidation consistently outperform random selection across the evaluated models, while semantic and structural similarity deliver more modest gains. Direct-utility selection produces improvements of up to 10% in the paper’s reported experiments.

This changes the inference decision. Once multiple candidate traces exist, the highest-value signal may not be “Which trace looks most like English reasoning?” but “Which trace contains steps that actually support the answer?”

That is a narrower and more defensible claim than saying English-based signals are bad. Semantic similarity remains competitive. It simply does not monopolize the quality signal.

The feature set is a control surface, not a complete theory

Sixteen manually specified features inevitably leave behaviors unmeasured. The paper therefore adds an automated discovery analysis: it learns sparse representations from trace chunks and inspects latent patterns most strongly associated with accuracy.

The purpose is exploratory extension, not a second independent thesis. The sparse-autoencoder analysis largely recovers patterns already visible in the hand-designed metrics while exposing additional behaviors.

The examples also show why language-specific diagnostics matter. In the reported Thai case, mixing multiple languages within the same reasoning trace has a -36% accuracy separation between examples where the concept is present versus absent. A Chinese MGSM-Rev2 concept involving explicitly ordered reasoning parts shows a +31% separation. These are concept associations in particular datasets and languages, not universal prescriptions.

For an evaluation team, that makes the 16-feature taxonomy more useful as an interpretable starting surface than as a finished ontology. Automated feature discovery can reveal recurring failure or success patterns that the original measurement design did not anticipate.

The business value is better diagnosis before better rewards

What the paper directly supports: multilingual trace-quality signals are not language invariant in magnitude, English similarity is useful but not uniformly dominant, and process-level utility measures can improve best-of-n selection on difficult mathematical reasoning tasks.

What Cognaptus infers for practice: multilingual evaluation teams should diagnose failures using several trace dimensions—translation quality, validity, dependency utility, consolidation, and language-specific behavior—before converting any one metric into a reward. Reward-model or post-training systems can then test whether weighting should vary by language rather than assuming an English-derived score transfers unchanged.

The same logic applies to inference routing. For high-value multilingual requests where extra inference cost is acceptable, process-level features can be tested as candidate-selection signals. This is more defensible than adopting them immediately as universal production rewards because the selection experiment holds the candidate set fixed and directly tests ranking performance.

Translation quality deserves separate attention. The paper finds it positively associated with reasoning accuracy, and its two benchmarks differ materially in translation procedure: MGSM-Rev2’s 250 queries are professionally translated, while the 60 AIME problems are machine-translated. For localized benchmark owners, translation is therefore part of the measurement system, not merely preprocessing.

The result does not justify a universal scoring recipe

Three boundaries constrain deployment.

First, the study covers mathematical reasoning only. Legal reasoning, commonsense tasks, multi-hop retrieval, or domain-specific workflows may produce different trace-quality relationships.

Second, the evidence comes from four open-weight reasoning models in the DeepSeek-R1-Distill-Qwen and Qwen3 families. Broader architectural transfer remains unresolved.

Third, best-of-32 selection buys ranking opportunity with additional compute. The appendix reports substantial increases in computation time as the candidate pool grows. A production system therefore needs to decide where the expected value of improved selection justifies repeated generation and re-ranking.

Several measurements also depend on GPT-4o annotation. Human verification is encouraging—majority-vote agreement averages 0.875 in French, 0.907 in Russian, and 0.938 in Chinese—but the check covers only a small MGSM-Rev2 subset with those three languages. It is evidence of annotation plausibility, not comprehensive multilingual validation.

The bounded conclusion is stronger than a universal recipe: treat multilingual reasoning quality as a set of measurable, language-sensitive signals. Validate which ones discriminate correctness in the target language and domain, then decide whether they belong in evaluation, reward design, or selective inference.

English similarity can remain one of those signals. It just should not receive universal authority by default.

Cognaptus: Automate the Present, Incubate the Future.