Seventeen Arabic Dialects, No Universal Speech-Recognition Winner

TL;DR for operators

A speech-recognition system ranked first across Arabic dialects may not be the strongest option for a particular regional market. The Almieyar benchmark evaluates 12 existing transcription systems on 13.7 hours of newly recorded speech across 17 dialects. GPT-4o-transcribe leads overall with 34.9% word error rate (WER), but Voxtral-Mini-4B records lower WER on Gulf Arabic, while SeamlessM4T-v2 leads on Egyptian Arabic.

The benchmark also reveals substantial differences between word-level and character-level recognition errors, making a single accuracy measure insufficient to characterize transcription quality. Its principal contribution is broader, culturally grounded evaluation of underrepresented Arabic varieties, including Ahwazi Arabic.

For teams selecting Arabic speech-to-text systems, these findings justify dialect-stratified testing, multiple error metrics, and representative regional acceptance criteria. The reported rankings remain preliminary evidence for deployment: each dialect contributes approximately 50 minutes of image-description speech, and the study does not measure conversational performance, latency, cost, or production reliability.

The best overall model may not serve every Arabic-speaking market best

Consider a company selecting one speech-to-text system for customers across several Arabic-speaking markets. Its evaluation team compares available models, identifies the lowest overall transcription error rate, and considers adopting that system everywhere.

That choice assumes the aggregate ranking reflects performance for the customers in each market. When regional speech varieties differ substantially, the assumption can break down.

The Almieyar benchmark, introduced by Ghahroodi and colleagues, provides a concrete example.¹ Among 12 evaluated speech-recognition systems, GPT-4o-transcribe achieves the lowest overall word error rate at 34.9%. Yet Voxtral-Mini-4B performs slightly better on Gulf Arabic, and SeamlessM4T-v2 performs better on Egyptian Arabic.

The results raise two separate evaluation questions: which dialects determine the model’s overall ranking, and whether counting incorrect words adequately describes the errors that users experience.

Almieyar expands which Arabic speech communities are evaluated

Existing Arabic speech-recognition benchmarks have often concentrated on Modern Standard Arabic or a limited selection of spoken dialects. The authors compare Almieyar with resources including MGB-2, which is predominantly based on broadcast speech containing Modern Standard Arabic, and Casablanca and NADI 2025, each covering eight dialects.

Almieyar covers 17 dialects across six families:

• Gulf: Bahraini, Omani, Qatari, Saudi, and Yemeni.
• Levantine: Jordanian, Lebanese, Palestinian, and Syrian.
• Maghrebi: Algerian, Libyan, Moroccan, and Tunisian.
• Iraqi/Ahwazi: Iraqi Arabic and Ahwazi Arabic.
• Egyptian Arabic.
• Sudanese Arabic.

The authors identify Ahwazi Arabic, associated with the Khuzestan region of Iran, as receiving its first published ASR benchmark representation.

The benchmark contains 13.7 hours of newly collected speech, approximately 50 minutes per dialect. Its primary contribution is not a new recognition model or a larger training corpus. It is an evaluation resource that broadens the communities and speech contexts against which existing systems can be tested.

Community participation shapes both recordings and reference transcripts

The collection procedure begins with native dialect-community coordinators selecting culturally relevant images across ten topics, including food, clothing, festivals, markets, architecture, and family gatherings.

Speakers describe these images through five structured scenarios addressing cultural context, principal subjects, surroundings, visual characteristics, and interpretation. Recordings generally last 30–60 seconds.

A Telegram-based collection interface generates an initial transcript using Google Cloud ASR. Speakers correct that transcript, after which community coordinators review dialect authenticity, spelling conventions, and code-switching.

This workflow has two methodological advantages.

First, culturally situated prompts elicit vocabulary and descriptions that may be missing from conventional broadcast-derived evaluation material. Second, newly recorded speech reduces the risk that identical recordings have already appeared in existing model-training corpora.

Neither advantage is a separately estimated causal effect. The study does not compare otherwise identical culturally grounded and conventional datasets, nor independently audit proprietary training corpora to establish complete freedom from contamination.

The overall ranking changes when dialect families are separated

The researchers evaluate 12 systems zero-shot, meaning the systems receive no additional training on Almieyar’s recordings. Model-specific standard inference settings are retained, and reference and generated Arabic transcripts undergo consistent text normalization.

Word error rate counts substitutions, deletions, and insertions relative to the number of words in a reference transcript. Lower values indicate fewer word-level transcription errors. A WER of 34.9% corresponds to approximately 35 word-edit errors per 100 reference words, not a guarantee that 65.1% of spoken sentences are correctly transcribed.

The central comparison is revealing.

Source: Almieyar, Table 2. All results are zero-shot WER percentages; lower is better. Bold indicates the lowest result within the displayed comparison.

GPT-4o-transcribe outperforms Voxtral-Mini-4B by 6.2 percentage points overall, yet its Gulf error rate is 0.7 percentage points higher. On Egyptian Arabic, SeamlessM4T-v2 improves on GPT-4o-transcribe by 5.8 percentage points despite having substantially worse overall performance.

These are observed benchmark differences, not statistically established advantages. The authors do not report uncertainty intervals, and the small samples limit confidence in closely separated rankings.

Variation also exists within dialect families. Appendix Table 4 reports Voxtral-Mini-4B WER ranging from 25% for Saudi Arabic to 43% for Bahraini Arabic, using rounded dialect-level results. That appendix covers 15 dialects and excludes GPT-4o-transcribe, so it cannot establish comparative per-dialect rankings against the overall-leading system.

For a regional voice product, the relevant model comparison therefore depends on the distribution of actual users, including variation within a nominal dialect family.

Word-level accuracy conceals different recognition errors

Choosing the right dialect-specific comparison is only part of the evaluation problem. The unit used to count errors also changes how transcription quality appears.

Almieyar reports three complementary measures:

Word error rate (WER) evaluates word substitutions, insertions, and deletions. It directly captures the failure to reproduce reference word sequences.

Character error rate (CER) measures corresponding errors at the character level. It can identify partial orthographic agreement even when entire words are scored as incorrect.

Match error rate (MER) uses an alternative word-event normalization that can moderate the influence of differences between generated and reference transcript lengths.

The clearest example comes from Wav2Vec2-53-AR, which records 79.1% WER but 29.4% CER overall.

That gap indicates that many incorrect word predictions still share substantial character-level structure with their references. It does not imply the system is operationally accurate: its word error rate remains far worse than those of the strongest evaluated models.

The authors connect some recognition failures to Arabic morphological structure, including weak-letter deletions and substitutions affecting short function morphemes. They also observe substitutions toward Modern Standard Arabic, although they do not identify these as the dominant pattern.

These linguistic explanations are qualitative interpretations of observed errors, not quantified causal decompositions.

MER supplies another perspective. Voxtral-Mini-4B records 41.1% overall WER and 37.8% MER. The MER results broadly retain the ordering among systems for which both measures are reported, while changing the numerical interpretation of errors.

Importantly, the published MER table contains 11 systems and excludes GPT-4o-transcribe. Its MER cannot be reconstructed from the reported WER or CER.

The measurement implication is that a model’s overall WER ranking and the nature of its transcription mistakes are different pieces of evidence. Applications requiring exact lexical content may assess these errors differently from applications where partial character agreement has some downstream value.

Regional acceptance tests need representative speech and explicit error criteria

Cognaptus inference: Almieyar’s strongest operational contribution is a better basis for designing model-selection tests, not a definitive vendor recommendation.

An Arabic-language voice-product team can apply its findings through three connected decisions.

First, define the communities the product actually serves. A regional acceptance test should contain speech from relevant dialects and speaker groups, rather than treating an overall Arabic-language score as sufficient evidence.

Second, establish which errors the application can tolerate. Word-level mistakes that alter names, requests, or other application-critical information may require separate review, even when aggregate character accuracy appears comparatively favorable. The paper does not measure these downstream consequences; they must be established using application-specific tests.

Third, evaluate candidate systems under deployment conditions. The benchmark supports an initial performance comparison, but an operational selection must also consider transcription latency, cost, privacy requirements, licensing, and deployment feasibility.

The broader methodological lesson is that evaluation data should reflect the linguistic communities and decisions for which a system will be accountable. Community participation helps construct that evidence, while error analysis determines what the resulting measurements actually establish.

What Almieyar cannot establish about production reliability

Several boundaries materially affect how the results should be used.

Speech modality. All recordings are structured descriptions of cultural images. They do not directly measure spontaneous customer conversations, broadcast speech, or medical, legal, and technical vocabulary.

Sample precision and representation. With approximately 50 minutes of speech per dialect, close model comparisons remain uncertain. Participants were predominantly young adults, gender representation varied across communities, and the headline results were not demographically stratified.

Reference ambiguity. Speaker correction and coordinator review improve reference quality, but dialectal spelling conventions and code-switching can still permit multiple valid transcriptions.

Adaptation and reproducibility. The evaluation establishes zero-shot performance only. It does not test dialect-specific fine-tuning or determine which systems would benefit most from adaptation. The source describes plans to release recordings, transcripts, scripts, and detailed results, without establishing that these resources were already publicly available.

Together, these constraints mean that the benchmark is better suited to identifying candidate systems and evaluation gaps than certifying a model for production use.

A useful benchmark identifies the conditions behind a ranking

Almieyar broadens Arabic speech-recognition evaluation through 17 dialects, community-guided speech collection, and a common comparison of 12 systems. Its findings show substantial remaining zero-shot transcription errors, changes in model leadership across dialect families, and meaningful differences between word-level and character-level measurements.

For operational teams, the central decision changes from selecting the model with the lowest aggregate error rate to determining which model meets defined quality requirements for the people and speech contexts being served.

The benchmark provides evidence for constructing that evaluation. Production acceptance still requires representative speech, application-specific error criteria, and testing under real operating conditions.

Cognaptus: Automate the Present, Incubate the Future.