The Same Math Score, Different Wrong Answers
TL;DR for operators A multilingual AI model can preserve nearly the same average accuracy after cultural localization while changing which individual questions it answers correctly. In DAR-MGSM, a benchmark covering Modern Standard Arabic and five Arabic dialects, researchers evaluate direct translations and culturally localized versions of mathematical word problems.1 Across three Gemma 4 models, 316 of 3,582 eligible translated–localized pairs (8.8%) switch between correct and incorrect answers, despite an average accuracy decline of only 1.2 percentage points. Across all 11 evaluated models, mean accuracy decreases from 65.5% to 63.4%, but model-level effects vary considerably. ...