Elamur Research⌕
☰
home / papers / 2023 / AraDiaWER: An Explainable Metric For Dialectical Arabic ASR

AraDiaWER: An Explainable Metric For Dialectical Arabic ASR

Paper · 2023 · Abdulwahab Sahyoun and Shady Shehata

Summary

Word error rate counts insertions, deletions and substitutions word by word. It ignores whether a mismatch is a trivial orthographic or clitic variant or a real change in meaning, which is a serious blind spot for Dialectical Arabic with its rich morphology. The authors propose AraDiaWER, a WER-based metric for Egyptian Arabic ASR that adds two interpretable scores. A syntactic score comes from CAMeLBERT-Mix morphological tagging (POS, lemma, clitics) compared by fuzzy string matching. A semantic score is the cosine similarity of MiniLM embeddings. The two scores are combined into an error weight that rescales WER. Perplexity-based fluency and UMAP projections of embeddings add further diagnostics. They test the metric on MGB-3 and the Egyptian part of FLEURS across hybrid systems (AALTO, TDNN), self-supervised models (Wav2Vec2 XLSR-53, HuBERT, Whisper) and their own fine-tuned AraDia-CTC. AraDiaWER averages 18.65% lower than WER. The authors present it as a transparent complement to WER, not a replacement, because it shows why a transcript is penalised.

Key points

  • The error weight W_err is the reciprocal of the sum of the semantic and syntactic scores, as the ablation numbers confirm. For example, AALTO has a syntactic score of 0.863 and a semantic score of 0.925, giving W_err of 0.559.
  • Syntactic tags are POS, lex and five clitic features (prc0–prc3, enc0). They come from the CAMeLBERT-Mix disambiguator and are refined by a unigram morphosyntactic analyzer.
  • Each tag is compared per word with a Levenshtein-based fuzzy ratio, then averaged over the sequence.
  • The ablation runs from syntax1 (POS only) to syntax7 (all tags). Adding tags raises the syntactic score and lowers W_err; for TDNN, W_err falls from 0.670 (syntax2) to 0.648 (syntax7).
  • Data:
    • MGB-3: 1,000 adaptation, 2,058 development and 297 test samples.
    • FLEURS Arabic: 428 Egyptian samples from 180 speakers.
    • AraDia-CTC: Wav2Vec2 with a CTC head, fine-tuned on Arabic speech from more than ten dialects.
  • RMSE between WER and AraDiaWER measures how far the new metric departs from WER. Perplexity and UMAP flag transcripts that score well on meaning and syntax yet are unintelligible.

Interesting ideas

  • I1. Instead of matching surface words, compare reference and hypothesis on parallel tag channels (POS, lemma, each clitic) with a fuzzy ratio, so you can see which layer of analysis diverged.
  • I2. Ablate the tag set step by step, adding one feature at a time, to see how each feature shifts the final score; this turns a metric into a diagnostic.
  • I3. Keep the classic metric intact and attach an explicit, decomposable multiplier built from interpretable sub-scores, so old benchmarks stay comparable while new insight is added.
  • I4. Use a fluency or perplexity check as a third, independent witness to catch outputs that look fine on content and grammar but are unreadable.
  • I5. Measure the RMSE between the new metric and the established one to quantify how much the new view actually changes the picture.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When scoring our parser or tagger against gold, we could report per-feature agreement (lemma, UPOS, each morphological feature) and see which layer breaks.
  • I2 (tokens): Applying the same incremental feature ablation to crate en's morphology output would show which features carry the most disagreement with treebank gold.
  • I3 (answer): Test scores on school-course or fable QA could keep raw accuracy but add a weight for answers that match in meaning but differ in form, reported separately.
  • I4 (back): Generated or compressor-reconstructed text could get a fluency check alongside graph-level fidelity, flagging round-trips that preserve meaning but produce awkward English.
  • I5 (learn): When we introduce a structured evaluation (e.g., frame-level match), reporting its divergence from LAS or exact match would show whether it adds information or merely restates the old score.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates138 works · 6,590 citations🎓Zayed UniversityAbu Dhabi · United Arab Emirates6 works · 67 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksSpeech & spoken language · 519 works

Cite

Cite as Abdulwahab Sahyoun and Shady Shehata. 2023. AraDiaWER: An Explainable Metric For Dialectical Arabic ASR. Proceedings of the Second Workshop on NLP Applications to Field Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.fieldmatters-1.8

@inproceedings{sahyoun2023aradiawer,
  title = {{AraDiaWER: An Explainable Metric For Dialectical Arabic ASR}},
  author = {Abdulwahab Sahyoun and Shady Shehata},
  booktitle = {Proceedings of the Second Workshop on NLP Applications to Field Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.fieldmatters-1.8},
  url = {https://doi.org/10.18653/v1/2023.fieldmatters-1.8}
}

References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Ali, Salam Khalifa, and Nizar Habash. 2019. Towards variability resistant dialectal speech evaluation: 20th Annual Conference of the International Speech Communication Association: Crossroads of Speech and Language, INTERSPEECH 2019. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2019-September:336–340. find
  2. Ahmed Ali, Walid Magdy, and Steve Renals. 2015. Multi-Reference Evaluation for Dialectal Speech Recognition System: A Study for Egyptian ASR. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 118–126, Beijing, China. Association for Computational Linguistics. find
  3. Ahmed Ali, Preslav Nakov, Peter Bell, and Steve Renals. 2017. WERd: Using Social Text Spelling Variants for Evaluating Dialectal Speech Recognition. Technical Report arXiv:1709.07484, arXiv. ArXiv:1709.07484 [cs] type: article. ↗
  4. Ahmed Ali and Steve Renals. 2020. Word Error Rate Estimation Without ASR Output: e-WER2. Technical Report arXiv:2008.03403, arXiv. ArXiv:2008.03403 [cs, eess] type: article. ↗
  5. Ahmed Ali, Yifan Zhang, Patrick Cardinal, Najim Dahak, Stephan Vogel, and James Glass. 2014. A complete KALDI recipe for building Arabic speech recognition systems. In 2014 IEEE Spoken Language Technology Workshop (SLT), pages 525–529. find
  6. Ahmed M. Ali and S. Renals. 2018. Word Error Rate Estimation for Speech Recognition: e-WER. In ACL. find
  7. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. arXiv:2006.11477 [cs, eess]. ArXiv: 2006.11477. ↗
  8. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2021. The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation. ArXiv:2106.03193 [cs]. ↗
  9. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. arXiv:2106.07447 [cs, eess]. ArXiv: 2106.07447. ↗
  10. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  11. Go Inoue, Salam Khalifa, and Nizar Habash. 2022. Morphosyntactic Tagging with Pre-trained Language Models for Arabic and its Dialects. ArXiv:2110.06852 [cs]. ↗
  12. Suyoun Kim, Abhinav Arora, Duc Le, Ching-Feng Yeh, Christian Fuegen, Ozlem Kalinli, and Michael L. Seltzer. 2021. Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding. ArXiv:2104.02138 [cs]. ↗
  13. Kathrein Abu Kwaik, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnika. 2018. A Lexical Distance Study of Arabic Dialects. Procedia Computer Science, 142:2–13. find
  14. Vimal Manohar, Daniel Povey, and Sanjeev Khudanpur. 2017. JHU Kaldi system for Arabic MGB-3 ASR challenge using diarization, audio-transcript alignment and transfer learning. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 346–352. find
  15. Maryam Najafian, Wei-Ning Hsu, Ahmed Ali, and James Glass. 2017. Automatic speech recognition of Arabic multi-genre broadcast media. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 353–359. find
  16. Alec Radford, John Kim, and Xu. 2022. Robust Speech Recognition via Large-Scale Weak Supervision. ↗
  17. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  18. Peter Smit, Siva Reddy Gangireddy, Seppo Enarvi, Sami Virpioja, and Mikko Kurimo. 2017. Aalto system for the 2017 Arabic multi-genre broadcast challenge. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 338–345. find
  19. Karel Vesely, Murali Karthick Baskar, Mireia Diez, and Karel Benes. 2017. MGB-3 but system: Low-resource ASR on Egyptian YouTube data. pages 368–373. find
  20. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. ArXiv:2002.10957 [cs]. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sahyoun-2023-aradiawer-explainable-metric-dialectical-arabic/