Model-Evaluation

TL;DR for operators UrduBench1 tests 23 open and open-weight models on 2,390 held-out Urdu questions spanning arithmetic reasoning, formal mathematics, commonsense, and knowledge tasks. The ranking gives little support to selecting an Urdu model from parameter count or a “reasoning” label alone: Gemma-3-12B-it leads the reported aggregate at 59.4%, while the larger reasoning-oriented DeepSeek-R1-Distill-Qwen-14B scores 44.9%. ...

TL;DR for operators A multilingual reasoning system may generate several plausible traces for the same request. The difficult part is deciding which trace to reward, trust, or return. The evidence here argues against using one English-derived trace-quality score as that decision rule. Across ten languages, many reasoning features are associated with correctness in broadly similar directions, but their effect sizes vary substantially and sometimes reverse. English semantic and structural similarity are generally positive signals, so the paper does not show that English-like reasoning is harmful. It shows that English similarity is not consistently the strongest signal. ...

TL;DR for operators A reasoning model can spend extra tokens reconsidering its answer and still fail to repair the mistake. Worse, unnecessary reconsideration can disturb reasoning that was already correct. The paper studied here treats that as a training problem rather than an invitation to add another inference-time review loop. It first filters a model’s own critiques using verifiable ground truth, then trains the model on the surviving examples. A second reinforcement-learning stage rewards both final-answer correctness and the quality of the reflective step itself. ...

TL;DR for operators A system generating several candidate answers eventually needs a ranking decision: which response should be shown, which should be discarded, and which should receive additional review. A reward model commonly reduces that decision to one scalar score derived from the prompt and response text. Oprea and Bâra test whether that score improves when the model is also given four explicit signals—response length, toxicity, refusal behavior, and prompt-response semantic similarity—and allowed to interpret those signals jointly with the text representation.1 On Anthropic HH-RLHF, the answer is consistently yes across ten evaluated model configurations. The strongest DeBERTa-v3 reward model moves from 0.74 to 0.84 ROC-AUC and from 0.72 to 0.83 pairwise accuracy. ...

TL;DR for operators A correct multimodal answer with citations can still have a weak evidence trail. In human evaluation on Video-MMMU, Gemini-3-Pro reached 86% question-answering accuracy but only 16.3 on MuRGAt-Score, meaning answer performance and fact-level grounding ranked very differently. MuRGAt evaluates two requirements separately. Coverage asks whether verifiable claims receive citations. Attribution asks whether the cited frames, audio, figures, or temporal segments are actually relevant to and sufficient for the factual claims they accompany. The combined score rewards systems only when both conditions hold. ...

TL;DR for operators A video system can produce the right answer without reliably understanding what happened over time. That becomes a procurement and QA problem when the decision depends on whether one event happened before another, how long something lasted, which direction it moved, whether an action repeated, or whether one event depended on another. ...

TL;DR for operators A model can receive a camera feed, spoken report, reference image, and text record without meaningfully reasoning across all four. In C$^3$PO1, 86–95% of observed failures across ten models in the paper’s failure analysis were classified as dominance-driven: one modality or prior drove the answer while other evidence was effectively ignored. ...

TL;DR for operators A model reads a chart, diagram, or photographed math problem and produces the wrong answer. Treating that event as a generic “reasoning failure” can send engineering effort to the wrong component. The model may have misread a number, attached a label to the wrong object, confused a scale or unit, or reasoned incorrectly after extracting the right facts. ...

TL;DR for operators A model can correctly recognize that one object is left of another and still fail when asked to generate that same relationship. More importantly, some visual weaknesses recur in both directions. AMVICC maps visual-language understanding and image generation onto visual concepts derived from the same underlying benchmark. Across the tested systems, weaknesses repeatedly appear in Quantity and Count, Positional and Relational Context, Orientation and Direction, and State and Condition. Text behaves differently: most tested multimodal language models avoid the paper’s Text failure threshold, while all three tested image generators fall below it for explicit generation. ...

TL;DR for operators A model can be given an explicit way to say “the available information is insufficient” and still choose an unsupported answer most of the time. Tahermazandarani, Mahmood, Islam, and Sheng test this directly across five LLMs.1 They remove the correct answer from medical multiple-choice questions, replace it with an insufficient-information option, and observe abstention rates ranging from just 0.156 to 0.382. Reported unsafe rates range from 0.186 to 0.828. In a separate experiment, progressively stronger warnings that the clinical information may be incomplete or ambiguous also produce little reduction in model confidence. ...