Elamur Research⌕
☰
home / papers / 2026 / When Does OmniASR Fail? A Fine-Grained Human Evaluation on Saudi Arabic Dialects

When Does OmniASR Fail? A Fine-Grained Human Evaluation on Saudi Arabic Dialects

Paper · 2026 · Hend Al-Khalifa

Summary

WER is the usual yardstick for speech recognition, but it gives every error the same weight and so hides which mistakes actually hurt users. This study examines Meta's multilingual OmniASR on Saudi Arabic dialectal speech, a setting complicated by diglossia, clitic-heavy morphology, code-switching and religious formulas. The author drew 103 clips from the SADA corpus, stratified by dialect, and ran them through OmniASR with default settings and no language conditioning. Three trained native-speaker annotators produced 264 judgments. Each judgment gave a 1–5 score for comprehensibility and for naturalness and tagged errors using a new 10-class taxonomy built for Arabic.

Quality was split into two camps. About a third of outputs were flawless, while roughly a fifth could not be used. Hallucinated and dropped words lowered comprehensibility far more than other errors. WER tracked human ratings about as closely as the annotators agreed with each other, yet it could not say which error types were responsible. The paper argues that Arabic ASR evaluation should account for error types alongside WER.

Key points

  • Data: 103 clips sampled from SADA (668 hours of Saudi TV speech covering Najdi, Hijazi and Gulf). The sample is 57.3% Najdi, with Gulf, Hijazi, Egyptian, Levantine and other varieties making up the rest. Clips run 4–19 words, about 10 on average.
  • The taxonomy has 10 categories:
    • generic: hallucination, deletion, semantic substitution
    • Arabic-specific: segmentation, morphology (clitics/conjugation), named entities, religious phrases, dialect↔MSA register substitution, code-switching, numbers
  • On average, OmniASR reached 42.2% WER and was rated 3.62/5 for comprehensibility. Of the outputs, 32.6% received perfect scores and 21.2% were essentially unusable.
  • Hallucinations cost −1.64 comprehensibility points and deletions −1.57. That is about six times the damage of named-entity errors.
  • WER correlated with comprehensibility at r = −0.679, close to pairwise annotator agreement (r = 0.61–0.67). The author notes several limitations: a modest sample, student annotators, a simple present/absent impact analysis instead of mixed-effects models, and only one system tested.

Interesting ideas

  • I1. Each error class is scored by the drop in the mean human rating when that class is present versus absent, which yields a ranking of error types by real harm instead of by frequency.
  • I2. Annotators heard the audio before seeing the system output, so their sense of the content was not shaped by the transcript.
  • I3. The automatic metric's correlation with humans is compared against how well humans agree with each other, which shows how much the metric can be trusted.
  • I4. Adequacy (comprehensibility) and fluency (naturalness) are rated separately, which exposes cases where an LLM-style decoder produces smooth text that is wrong, called the "fluency trap" in the limitations.
  • I5. Dropping a single negation word ("I do NOT agree" becoming "I agree") counts as one WER error but reverses the meaning, which is why deletions are so damaging.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Tag parser and frame errors by type and measure how much each type lowers answer accuracy on downstream tests, so repair rules target the most damaging types first.
  • I2 (learn): When an LLM teacher or human checks a parse, have them produce their own reading of the sentence before seeing Mova's tree, so the check does not simply defer to Mova's output.
  • I3 (learn): Compare LAS and frame-level scores against agreement between witness parsers or human judges, so we can tell whether a metric is already near the ceiling.
  • I4 (back): Score generated and compressed text separately for faithfulness to the meaning graph and for fluency, so polished but unfaithful outputs are caught.
  • I5 (frames): Add a polarity and negation consistency check to frames, and weight negation-loss errors heavily when evaluating the stages that feed them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Saud UniversityRiyadh · Saudi Arabia37 works · 527 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Hend Al-Khalifa. 2026. When Does OmniASR Fail? A Fine-Grained Human Evaluation on Saudi Arabic Dialects. Proceedings of Speech Language Models in Low-Resource Settings: Performance, Evaluation, and Bias Analysis (SPEAKABLE) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2wmxhnwmucq4

@inproceedings{alkhalifa2026when,
  title = {{When Does OmniASR Fail? A Fine-Grained Human Evaluation on Saudi Arabic Dialects}},
  author = {Hend Al-Khalifa},
  booktitle = {Proceedings of Speech Language Models in Low-Resource Settings: Performance, Evaluation, and Bias Analysis (SPEAKABLE) @ LREC 2026},
  year = {2026},
  doi = {10.63317/2wmxhnwmucq4},
  url = {https://doi.org/10.63317/2wmxhnwmucq4}
}

References

14 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ali, A., Bell, P., Glass, J., Messaoui, Y., Mubarak, H., Renals, S., & Zhang, Y. (2016). The MGB-2 Challenge: Arabic Multi-Dialect Broadcast Media Recognition. In Proceedings of the 2016 IEEE Spoken Language Technology Workshop (SLT) , pp. 279-284. IEEE. https://doi.org/10.1109/SLT.2016.7846277 ↗
  2. Alharbi, S., Alowisheq, A., Tüske, Z., Darwish, K., Alrajeh, A., Alrowithi, A., Bin Tamran, A., Ibrahim, A., Aloraini, R., Alnajim, R., Alkahtani, R., Almuasaad, R., Alrasheed, S., Alsubaie, S., & Alonaizan, Y. (2024). SADA: Saudi Audio Dataset for Arabic. In Proceedings of ICASSP 2024 - IEEE International Conference on Acoustics, Speech and Signal Processing , pp. 10286-10290. IEEE. https://doi.org/10.1109/ICASSP48485.2024.10 446243 ↗
  3. Badawi, E. S. (1973). Mustawayat al-'arabiyya al-mu'asira fi Misr [Levels of Contemporary Arabic in Egypt]. Cairo: Dar al-Ma'arif. find
  4. Belinkov, Y., Barrón-Cedeño, A., Magidow, A., Shmidman, A., & Romanov, M. (2019). Studying the history of the Arabic language: Language technology and a large-scale historical corpus. Language Resources and Evaluation , 53(4), 771-805. https://doi.org/10.1007/s10579-019-09460-w ↗
  5. Boudelaa, S., & Marslen-Wilson, W. D. (2013). Morphological structure in the Arabic mental lexicon: Parallels between standard and dialectal Arabic. Language and Cognitive Processes , 28(10), 1453-1473. https://doi.org/10.1080/01690965.2012.71962 9 ↗
  6. Ferguson, C. A. (1959). Diglossia. Word , 15(2), 325-340. https://doi.org/10.1080/00437956.1959.116597 02 ↗
  7. Holes, C. (2004). Modern Arabic: Structures, Functions, and Varieties (Revised ed.). Georgetown University Press. Keren, G., Kozhevnikov, A., Meng, Y., Ropers, C., Setzler, M., Wang, S., Adebara, I., Auli, M., Balioglu, C., Chan, K., Cheng, C., Chuang, J., Droof, C., Duppenthaler, M., Duquenne, P.-A., Erben, A., Gao, C., Gonzalez, G. M., Lyu, K., ... find
  8. Yates, S. (2025). Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages. arXiv preprint arXiv:2511.09690 . https://arxiv.org/abs/2511.09690 ↗
  9. Koehn, P., & Monz, C. (2006). Manual and Automatic Evaluation of Machine Translation between European Languages. In Proceedings of the Workshop on Statistical Machine Translation , pp. 102-121. Association for Computational Linguistics.
  10. Martindale, M. J., & Carpuat, M. (2018). Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MT. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (AMTA) , Vol. 1, pp. 13-25. ↗
  11. McCowan, I., Moore, D., Dines, J., Gatica-Perez, D., Flynn, M., Wellner, P., & Bourlard, H. (2004). On the Use of Information Retrieval Measures for Speech Recognition Evaluation. IDIAP Research Report 04-73 . IDIAP. find
  12. Saiegh-Haddad, E. (2018). MAWRID: A Model of Arabic Word Reading in Development. Journal of Learning Disabilities , 51(5), 454-462. https://doi.org/10.1177/0022219417720460 ↗
  13. Tatman, R. (2017). Gender and Dialect Bias in YouTube's Automatic Captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing , pp. 53-59. Association for Computational Linguistics. https://doi.org/10.18653/v1/W17-1606 ↗
  14. Wang, Y., Alhmoud, A., & Alqurishi, M. (2024). Open Universal Arabic ASR Leaderboard. https://arxiv.org/abs/2412.13788 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khalifa-2026-when-does-omniasr-fail-fine/