Elamur Research⌕
☰
home / papers / 2025 / Towards stable AI systems for Evaluating Arabic Pronunciations

Towards stable AI systems for Evaluating Arabic Pronunciations

Summary

Self-supervised speech models such as wav2vec 2.0 transcribe words and sentences well but do poorly when the input is a single spoken Arabic letter. A lone letter lasts only a fraction of a second, has no neighbouring sounds that shape it, and carries no lexical context. Emphatic and pharyngeal consonants make the task harder still. The authors built "Horouf", a corpus of about ten thousand recordings of diacritised isolated letters, collected through their own web and Android apps from native and non-native speakers. Off-the-shelf wav2vec 2.0 reaches roughly 35% accuracy on it. A small MLP trained on wav2vec embeddings raises this to about 65%. That MLP is fragile: a tiny amplitude perturbation (ϵ = 0.05) pushes it back to about 32%. PGD-based adversarial training limits the loss under noise to about 9 points and keeps clean accuracy. The work matters for pronunciation tutoring, speech therapy and Tajweed checking, where each individual letter has to be judged.

Key points

  • Each label is a letter–diacritic pair (fatha /a/, kasra /i/, damma /u/, sokoon), which gives 112 classes in total.
  • The Horouf dataset has about 10k recordings with speaker metadata (gender, age, nativeness, continent). Annotation was done by experts, and the data and code are available on request.
  • Reported accuracies:
    • pretrained wav2vec 2.0: about 35%;
    • MLP on wav2vec embeddings: about 65% on clean audio;
    • the same MLP at ϵ = 0.05: about 32%.
  • Under PGD attacks of growing strength (ϵ up to 0.15), the vanilla MLP nearly collapses. The adversarially trained MLP with the same architecture declines only modestly.
  • Planned future work: combine a grapheme-level segmenter with the letter classifier to score each letter inside whole words. Example: telling "abi" apart from the learner error "api".

Interesting ideas

  • I1. Testing a strong model on minimal units with all context removed exposes weaknesses that context normally hides. — For Mova (tokens): Run the tagger and lemmatiser on isolated words or very short fragments to see which decisions depend on context and which come from the word itself.
  • I2. A curve of accuracy against perturbation strength (clean, then increasing ϵ) is a more honest measure than a single clean score. — For Mova (learn): Plot parser and frame accuracy against controlled text noise levels (typos, casing, punctuation loss) and report this next to the negative controls.
  • I3. Adversarial training feeds the model the worst-case input inside a bounded neighbourhood, so stability is learned rather than hoped for. — For Mova (tree): When training the perceptron parser or gparse, add the token or embedding perturbations that most hurt the current model, with a strict bound on how much they change the input.
  • I4. A light classifier head on frozen pretrained embeddings nearly doubled accuracy over the end-to-end model on a niche task. — For Mova (tree): Small probing heads on frozen gparse representations could cover rare UD relations or dialect conversions without retraining the whole transformer.
  • I5. Segment the input, give each unit its own confidence, then combine them into an overall score, so feedback points at the exact faulty piece. — For Mova (answer): When grading student answers or maths solutions, score each step or clause separately and show which one failed instead of returning a single pass/fail verdict.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2508.19587 · arXiv 2508.19587

@misc{zaatiti2025stable,
  title = {{Towards stable AI systems for Evaluating Arabic Pronunciations}},
  author = {Zaatiti, Hadi and Hajri, Hatem and Abdullah, Osama and Masmoudi, Nader},
  year = {2025},
  doi = {10.48550/arxiv.2508.19587},
  eprint = {2508.19587},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2508.19587}
}

References

13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eiad Almekhlafi, Moeen Al-Makhlafi, Erlei Zhang, Jun Wang, and Jinye Peng. A classification benchmark for arabic alphabet phonemes with diacritics in deep neural networks. Computer Speech & Language, 71:101274, 2022. find
  2. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: a framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA, 2020. Curran Associates Inc. find our card
  3. Lila Kim and Cédric Gendrot. Using wav2vec 2.0 for phonetic classification tasks: Methodological aspects. In Proceedings of Interspeech 2024, 2024. to appear. find
  4. Norah Alrashoudi, Hend Al-Khalifa, and Yousef Alotaibi. Improving mispronunciation detection and diagnosis for non-native learners of the arabic language. Discover Computing, 28(1):1, 2025. find
  5. Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech-to-text. In 2018 IEEE Security and Privacy Workshops (SPW), pages 1–7. IEEE, 2018. find
  6. Ngoc Dung Huynh, Mohamed Reda Bouadjenek, Imran Razzak, Kevin Lee, Chetan Arora, Ali Hassani, and Arkady Zaslavsky. Adversarial attacks on speech recognition systems for mission-critical applications: A survey. arXiv preprint arXiv:2202.10594, 2022. ↗
  7. Jonatas Grosman. wav2vec2-large-xlsr-53-arabic. https://huggingface.co/jonatasgrosman/wav2vec2-large-xlsr-53-arabic, 2021. ↗
  8. Yun Tang, Guohong Ding, Jing Huang, Xiaodong He, and Bowen Zhou. Deep speaker embedding learning with multi-level pooling for text-independent speaker verification. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6116– 6120, 2019. find
  9. David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. X-vectors: Robust dnn embeddings for speaker recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5329–5333, 2018. find
  10. Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross-lingual representation learning for speech recognition. In Interspeech 2021, pages 2426–2430, 2021. find
  11. Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141–142, 2012. find
  12. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. find our card
  13. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. Authors Hadi Zaatiti received the M.E. degree in System Architecture and Embedded Systems from CentraleSupélec, Paris, France, in 2015, and the Ph.D. degree in Computer Science from the French Atomic Energy Commission in 2018. He worked on modeling and formal verification of hybrid dynamical systems at the French Atomic Energy Commission from 2015 to 2018. From 2018 to 2022, he was a System Architect and Data Scientist at the Institute of Research and Technology SystemX. He then joined Airbus Defence and Space as a System Engineer from 2022 to 2024. Since 2024, he has been a Research Scientist at New York University Abu Dhabi, leading MEG projects within the Biomedical Imaging Core. Hatem Hajri received his M.S. and Ph.D. degrees in Applied Mathematics from Paris-Sud University, France, in 2008 and 2011, respectively. From 2011 to 2016, he held teaching and research positions at Paris Nanterre University (Paris 10), the University of Luxembourg, and the University of Bordeaux, where he worked on various topics in stochastic calculus and statistical learning on manifolds. Between 2016 and 2018, he was a researcher at the VeDeCoM Institute in Versailles, focusing on artificial intelligence and autonomous driving. Since 2018, he has been with the Institute for Technological Research SystemX, serving as a lead research scientist on trustworthy AI. Osama Abdullah holds a Ph.D. in Bioengineering from the University of Utah, where his research focused on diffusion tensor MRI of the myocardium. He also holds an M.S. in Bioengineering from the University of Illinois at Chicago and a B.Sc. in Electrical Engineering from the University of Jordan. Since 2017, he has served as an MRI Physicist and Research Instrumentation Scientist at the Bio-Medical Imaging Core of New York University Abu Dhabi, where he applies advanced multi-modal imaging to study brain disorders such as tumors, visual perception dysfunctions, and multiple sclerosis. He also develops imaging analysis tools and safety protocols. Previously, he managed the small animal imaging core at the University of Utah. Dr. Abdullah has received several honors, including the Young Investigator Award from the Department of Education and Knowledge (2020) and merit awards from the ISMRM in 2013 and 2016 for his contributions to MRI research. Nader Masmoudi is a Professor of Mathematics at the Courant Institute of Mathematical Sciences at New York University. His research centers on partial differential equations, nonlinear analysis, and mathematical fluid dynamics, with landmark contributions to the theory of the Navier–Stokes equations, stability of fluid interfaces, and related models in physics. He did his bachelor at the Ecole Normale Supérieure in 1996 and received his Ph.D for Paris-Dauphin in 1999. Professor Masmoudi has held visiting and affiliated positions at leading institutions and has been recognized with several prestigious honors, including election to the American Academy of Arts and Sciences in 2021, the Kuwait prize 2020, the king Faisal prize 2022, the Fermat prize. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaatiti-2025-towards-stable-ai-systems-evaluating/