English Looks Ready. Amharic Says Otherwise: What ADAGE Exposes in Multilingual Evaluation

TL;DR for operators

A multilingual model can look ready on an English reasoning benchmark and still perform close to chance in a strategically important native language. In the reported zero-shot evaluation, Gemma 3 27B scores 83.0% on English ePiC, 70.1% on Arabic CAPR, 41.3% on Amharic CAPR, and 86.0% on Japanese CAPR.

ADAGE addresses this measurement problem by constructing native-language reasoning tests rather than translating English questions. Its three CAPR benchmarks use culturally native proverbs, newly generated target-language scenarios, and structured wrong answers designed to test whether a model can apply an underlying meaning rather than match topics or wording.

For deployment teams, the implication is specific: an English or translated multilingual score is insufficient evidence for launch readiness in a target market. Native evaluation should enter model selection and pre-launch validation. But the Amharic results also expose a second problem: the LLMs used to validate the benchmark are themselves less reliable in the low-resource language, so automated evaluation requires human oversight where evaluator competence is uncertain.

A model can pass English and still fail the market

Consider a product team choosing one multilingual model for several language markets. The model performs well on an established English reasoning benchmark. The team translates its evaluation suite, sees broadly acceptable results, and assumes it has tested the same capability elsewhere.

That assumption becomes difficult to maintain when the same model produces radically different outcomes across native-language tests. Gemma 3 27B reaches 83.0% on the English ePiC proverb-reasoning benchmark, but 70.1% on Arabic, 41.3% on Amharic, and 86.0% on Japanese under the study’s zero-shot protocol. CAPR has four answer choices, so chance performance is 25%. A 41.3% result is therefore not merely a modest decline from English performance.

Ahmed Haj Ahmed and Alvin Grissom’s ADAGE paper¹ is built around this evaluation problem. The relevant capability is analogical application: recognizing the underlying lesson or relationship expressed by a proverb and applying it to a new situation with different surface wording. A translated benchmark can change the language while preserving the source culture’s original reasoning structure. That makes it difficult to tell whether the resulting score reflects target-language competence, cultural familiarity, reasoning, or artifacts of translation.

ADAGE builds native tests instead of translating English ones

ADAGE constructs the test from native-language material. The authors instantiate the pipeline with Arabic proverbs from Jawaher, Amharic proverbs from ProverbEval, and Japanese proverbs from Kotowaza.org.

The construction process has seven stages: source acquisition, thematic clustering, scenario generation, automated filtering, distractor selection, multi-agent validation, and dataset curation. Each proverb receives candidate modern scenarios generated directly in the target language. Filters then remove candidates with problems such as insufficient target-script coverage, excessive lexical overlap with the proverb, unsuitable length, or excessive embedding similarity.

The benchmark also attempts to make wrong answers informative. Each four-choice item contains the correct proverb plus three types of distractor: one from the same thematic cluster, one semantically nearby but from another theme, and one from a random cluster. The purpose is diagnostic. A model that understands only the broad topic should have more difficulty separating the correct proverb from a plausible thematic alternative than from an unrelated one.

The scale of the resulting datasets varies substantially. CAPR-ar contains 1,222 test items, CAPR-am 716, and CAPR-ja only 100. The pipeline is therefore common across languages, but the resulting benchmark instances are not equally powered or equally comprehensive.

The error pattern reveals more than the headline score

The strongest observed CAPR results are 70.1% for Arabic, 41.3% for Amharic, and 93.0% for Japanese. The paper also reports that models performing strongly on English ePiC can lose roughly 12 to 52 percentage points on the native benchmarks.

Those score gaps establish the measurement problem, but the distractor analysis helps interpret it.

Arabic errors show a clearer difficulty structure: cross-theme near-matches attract 34.4% of errors, same-cluster distractors 31.2%, and random-cluster distractors 26.9%. Amharic is much flatter at 32.9%, 34.7%, and 32.4%.

That flat distribution changes how a low Amharic score should be read. It is not sufficient evidence that a model possesses adequate Amharic competence but lacks abstract reasoning ability. The authors instead interpret Amharic language competence as a possible binding constraint: if the model cannot reliably process the language, adding reasoning capacity may have little opportunity to help.

Amharic shows why parameter count is an incomplete procurement metric

The scaling results reinforce that interpretation. Arabic exhibits clearer gains as models become larger, while Amharic remains weak across much of the tested range. The paper also reports position bias among some larger models in Amharic and mean cross-model item-level agreement near zero.

Training composition appears relevant as well. Aya Expanse 8B scores 53.5% on Arabic, compared with 44.4% for Aya 8B, a 9.1 percentage-point improvement. The authors note that this gain is comparable in magnitude to substantial scaling within the Gemma family.

The paper does not causally isolate training data from architecture, model family, parameter count, or other differences. The business inference is narrower: when a target language is poorly represented, buying a larger general model may produce less value than selecting or training a model with stronger language-specific coverage. Parameter count alone is therefore a weak basis for multilingual model procurement.

The evaluator can share the model’s language weakness

ADAGE also exposes a governance problem inside the evaluation process.

Three open-weight LLM judges score benchmark items for faithfulness, naturalness, and distractor plausibility. Arabic and Japanese achieve majority-vote pass rates of at least 98.7% across these criteria. Amharic naturalness passes at only 77.7%, and 35.3% of Amharic items are flagged, versus 4.4% for Arabic and 0.7% for Japanese.

This does not mean the Amharic benchmark result collapses. The authors perform a robustness check by removing flagged items and re-evaluating leading models. Accuracy changes by no more than 1.3 percentage points. That sensitivity test supports the conclusion that the broad cross-language performance pattern is not being created by flagged items.

It does expose something else: automated evaluation infrastructure inherits language-resource constraints. If the judge model is weak in the same language as the evaluated model, an automated validation pipeline can become least trustworthy precisely where stronger scrutiny is needed.

For a product team entering a lower-resource market, native-speaker review is therefore not merely a content-quality enhancement. It is part of evaluator validation.

Use native evaluation as a deployment gate, not a universal intelligence score

The paper directly supports a targeted change in multilingual evaluation practice: construct or acquire tests written from native cultural and linguistic material, and run them before concluding that performance on English or translated benchmarks transfers to a target market.

For model-selection teams, that means comparing candidates on the languages that actually determine deployment success. For evaluation teams, the structured distractors offer information beyond aggregate accuracy: they can reveal whether errors respond to semantic difficulty or remain effectively undifferentiated. For teams allocating training resources, the Arabic results suggest that multilingual data composition deserves consideration alongside scale.

The boundaries are equally concrete. ADAGE covers three CAPR languages and one form of culturally grounded reasoning: applying proverb meanings to new scenarios. Models are evaluated zero-shot, are open-weight, and are no larger than 35B parameters. Japanese has only 100 test items. The English ePiC comparator also differs from CAPR because it has five rather than four answer options and uses crowdsourced rather than LLM-generated scenarios.

The study therefore does not estimate a causal “translation penalty,” nor does it establish a general ranking of multilingual intelligence. What it provides is a stronger diagnostic question for deployment: has the capability actually been tested in the language and cultural setting where the product will operate?

A model that scores well in English has demonstrated something valuable. ADAGE shows why a multilingual launch decision requires evidence closer to the market itself.

Cognaptus: Automate the Present, Incubate the Future.