Reasoning Labels Don’t Travel: What UrduBench Changes About Model Selection

TL;DR for operators

UrduBench¹ tests 23 open and open-weight models on 2,390 held-out Urdu questions spanning arithmetic reasoning, formal mathematics, commonsense, and knowledge tasks. The ranking gives little support to selecting an Urdu model from parameter count or a “reasoning” label alone: Gemma-3-12B-it leads the reported aggregate at 59.4%, while the larger reasoning-oriented DeepSeek-R1-Distill-Qwen-14B scores 44.9%.

The more useful pattern is workload dependence. Reasoning-specialized models are competitive or strong on mathematics, especially harder MATH-500 problems, but they do not dominate commonsense or knowledge questions. Prompting also changes results materially, and adding few-shot reasoning examples can reduce accuracy for some models.

For procurement, evaluate the candidate model on the intended Urdu workload, with the prompting policy you expect to deploy. Add language stability as a separate quality measure rather than assuming an accurate answer will also remain consistently in Urdu. The evidence supports comparative selection; it does not isolate whether scale, multilingual training, reasoning supervision, or language consistency causes the observed differences.

The strongest Urdu model is not the largest reasoning model

Suppose a product team has narrowed an Urdu deployment to several plausible LLMs. One is larger. Another is explicitly trained for reasoning. A third has stronger multilingual credentials. Those attributes are reasonable screening signals, but UrduBench shows that they are poor substitutes for task-level evaluation.

The clearest example is the aggregate ranking. Gemma-3-12B-it reaches 59.4% across the benchmark’s selected aggregate conditions. DeepSeek-R1-Distill-Qwen-14B, despite being larger and reasoning-oriented, reaches 44.9%. Gemma-3-4B-it also scores 44.9%, ahead of several 7B and 8B alternatives.

That does not establish that 12B models are better than 14B models, or that instruction tuning beats reasoning training. The comparison is descriptive, not causal. It establishes something more operationally useful: model size and product category do not determine the Urdu ranking reliably enough to serve as procurement rules.

Different Urdu workloads produce different winners

UrduBench is an Urdu adaptation of four established benchmarks rather than a newly authored problem set. It covers 250 MGSM items, 500 MATH-500 problems, 1,140 CommonSenseQA questions, and 500 OpenBookQA questions.

That mix matters because the model ordering changes with the work.

On MATH-500, reasoning-oriented models become much more competitive. Phi-4-mini-reasoning reaches 65.8%, DeepSeek-R1-Distill-Qwen-7B reaches 57.0%, and the 14B DeepSeek variant reaches 69.2%. Gemma-3-12B-it remains narrowly ahead at 70.0%.

The higher-difficulty results sharpen the distinction. Gemma-3-12B-it drops from 86.1% at level L1 to 48.5% at L5. DeepSeek-R1-Distill-Qwen-14B moves from 79.1% to 63.4%. The appendix difficulty table is therefore best read as a sensitivity test of mathematical robustness: it shows that nearly equal overall mathematics scores can conceal different degradation profiles as problems become harder.

The same reasoning specialization does not transfer uniformly to other tasks. DeepSeek-R1-Distill-Qwen-14B records 19.4% on CommonSenseQA and 31.8% on OpenBookQA, while Falcon-h1-7B-Instruct reaches 59.3% and 32.8% respectively.

For an Urdu tutoring system dominated by formal mathematics, the procurement ranking may therefore look different from one for customer support, general assistance, or knowledge-heavy question answering.

Prompt policy is part of the model configuration

The paper also tests the same MGSM arithmetic workload under direct answers, explicit intermediate reasoning, and few-shot examples combined with intermediate reasoning.

Explicit reasoning usually improves MGSM performance relative to direct prompting, but the effect is model-dependent. Gemma-3-4B-it rises from 54.8% under direct prompting to 64.0% with Chain-of-Thought. Yet its few-shot-plus-reasoning score falls to 44.0%.

Other models show different trajectories. Qwen2.5-7B-Instruct moves from 53.2% to 57.6% and then 60.4%. MiMo-7B-Base falls from 29.6% to 20.8% before jumping to 56.4% with few-shot examples.

The practical consequence is not that one prompt style is superior. It is that the prompt should be treated as part of the evaluated deployment configuration. A leaderboard result obtained under one reasoning policy cannot safely be transferred to another without testing.

Translation quality is part of measurement quality

Translated benchmarks can become unreliable when the question and its answer choices lose shared context or when translation errors change what is being tested. UrduBench addresses this before model evaluation.

For multiple-choice tasks, the pipeline translates each question together with all answer choices rather than translating fragments independently. It generates candidates using IndicTrans2, NLLB, Qwen-3-30B, and Gemini-2.5-Pro; GPT-5.1 then compares the candidates against the English source and selects or synthesizes their strengths. Structural and Urdu-language heuristics reject malformed outputs, and professional Urdu-speaking annotators perform final selection and correction.

Human candidate-selection statistics strongly favor Gemini-2.5-Pro among the four raw translation systems—for example, 826 selections on CommonSenseQA versus 74 for IndicTrans2, 52 for NLLB-200, and 17 for Qwen-3-30B. That table is evidence about the relative quality of raw candidates under human review. It does not, by itself, prove that the full ensemble pipeline outperforms every single-system alternative.

For organizations localizing their own internal evaluation sets, that distinction matters. Translation is part of the measurement system. If localization changes the question, a precise model score can still measure the wrong thing.

Accuracy and staying in Urdu are separate checks

A model can solve a problem while drifting into another language. UrduBench therefore measures answer-language consistency after removing special tokens and mathematical notation and excluding responses too short for reliable language identification.

Several models reach 100% consistency on the reported MGSM Chain-of-Thought test, including Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Gemma-3-12B-it. DeepSeek-R1-Distill-Qwen-14B records 82.0%.

The paper reports an association between higher Urdu consistency and stronger MGSM reasoning performance, but this analysis covers a selected set of models under one benchmark and prompting condition. It does not establish that suppressing code-switching would itself improve reasoning.

For deployment, the safer interpretation is to measure both. Accuracy answers whether the model completed the task correctly. Language consistency answers whether it delivered that capability in the user-facing language required by the product.

A procurement scorecard should replace model-label heuristics

The paper supports a straightforward evaluation design for Urdu-facing systems:

Cognaptus inference: a team choosing an Urdu model should benchmark the configuration it will actually deploy—model, task mix, prompting policy, and language requirements—rather than purchase capability by proxy.

What remains uncertain is why individual models rank where they do. The study does not experimentally isolate multilingual pretraining, Urdu exposure, parameter scale, reasoning distillation, or language stability. Those variables are plausible explanations, not identified causal mechanisms.

The benchmark changes what should be measured before deployment

UrduBench’s main contribution is not a new universal ranking. It provides a more disciplined way to discover when familiar model-selection shortcuts stop working in Urdu.

The strongest aggregate model is not simply the largest reasoning model. The strongest mathematical model need not be the strongest commonsense model. Additional reasoning examples do not reliably improve every architecture. And an accurate response can still fail a product requirement by leaving the target language.

For Urdu-facing deployments, selection should therefore move closer to the production workload: benchmark the relevant tasks, test the intended prompt, inspect performance at the difficulty levels that matter, and measure whether the model remains in Urdu. That produces a procurement decision grounded in observed behavior rather than labels whose meaning changes across languages and workloads.

Cognaptus: Automate the Present, Incubate the Future.