Elamur Research⌕
☰
home / papers / 2026 / HARNESS: Lightweight Distilled Arabic Speech Foundation Models

HARNESS: Lightweight Distilled Arabic Speech Foundation Models

Paper · 2026 · Vrunda Nileshkumar Sukhadia and Shammur Absar Chowdhury

Summary

Large self-supervised speech models work well but are too heavy for many real deployments. Arabic is also poorly served by generic multilingual encoders because of its strong dialectal variation and frequent English or French borrowing. The authors present HArnESS, a HuBERT-style model family built for Arabic and trained from scratch on mixed Arabic-English audio. English data adds acoustic diversity and reflects code-switching. A 24-layer teacher, HArnESS-L, is refined through masked prediction of K-means pseudo-labels, with each iteration learning from the previous model's clustered embeddings. Later iterations compress it into a shallower student (HArnESS-S) and a shallower, narrower one (HArnESS-ST). The authors also try PCA on teacher embeddings before clustering, which gives small students simpler targets. On ASR, dialect identification and speech emotion recognition, HArnESS outperforms HuBERT-Large and XLS-R, and the compact students stay competitive despite heavy size reduction. The released models offer a practical Arabic speech backbone for settings with limited compute.

Key points

  • Pretraining data: roughly 4K hours of Arabic audio combined with 3.56K hours of English audio.
    • Sources include QASR, MGB3, LibriSpeech, Common Voice and GigaSpeech, plus YouTube content from 15 Arabic-speaking countries.
    • Augmentation raises the total to roughly 23K hours, mostly via SpecAugment, with about 300 hours of added background noise.
  • Pseudo-label schedule:
    • Iteration 1 clusters MFCC features.
    • Iteration 2 clusters 9th-layer embeddings.
    • Later iterations use last-layer outputs.
    • The architecture stays fixed for two iterations, and compression (depth, width, attention heads) begins from the third.
  • Training choices: the loss covers both masked and unmasked frames for stability. Students are initialized either randomly or by averaging blocks of teacher layers.
  • Supervision compression: PCA shrinks the teacher's target space rather than the student's input, which is meant to help thin students.
  • Release: the distilled models and benchmark resources are available at huggingface.co/QCRI/distillHarness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksEfficient models & compression · 242 works

Cite

Cite as Vrunda Nileshkumar Sukhadia and Shammur Absar Chowdhury. 2026. HARNESS: Lightweight Distilled Arabic Speech Foundation Models. Proceedings of Speech Language Models in Low-Resource Settings: Performance, Evaluation, and Bias Analysis (SPEAKABLE) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2x7bxm8g49ju

@inproceedings{sukhadia2026harness,
  title = {{HARNESS: Lightweight Distilled Arabic Speech Foundation Models}},
  author = {Vrunda Nileshkumar Sukhadia and Shammur Absar Chowdhury},
  booktitle = {Proceedings of Speech Language Models in Low-Resource Settings: Performance, Evaluation, and Bias Analysis (SPEAKABLE) @ LREC 2026},
  year = {2026},
  doi = {10.63317/2x7bxm8g49ju},
  url = {https://doi.org/10.63317/2x7bxm8g49ju}
}

References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ali Abouzeid, Bilal Elbouardi, Mohamed Maged, and Shady Shehata. 2025. Arabemonet: A lightweight hybrid 2d cnn-bilstm model with attention for robust arabic speech emotion recognition.
  2. Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang. 2019. The mgb-2 challenge: Arabic multi-dialect broadcast media recognition. find
  3. Ahmed Ali, Shammur Chowdhury, Mohamed Afify, Wassim El-Hajj, Hazem Hajj, Mourad Abbas, Amir Hussein, Nada Ghneim, Mohammad Abushariah, and Assal Alqudah. 2021. Connecting Arabs: bridging the gap in dialectal speech recognition. Communications of the ACM, pages 124–129. ↗
  4. Takanori Ashihara, Takafumi Moriya, Kohei Matsuura, and Tomohiro Tanaka. 2022. Deep versus wide: An analysis of student architectures for task-agnostic knowledge distillation of self-supervised speech models. In 23rd Annual Conference of the International Speech Communication Association, INTERSPEECH 2022. find
  5. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021. Xls-r: Self-supervised cross-lingual speech representation learning at scale. In Proceedings of Interspeech.
  6. Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. 2022. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pages 1298–1312. PMLR. ↗
  7. Heng-Jui Chang, Shu-wen Yang, and Hung-yi Lee. 2022. Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7087–7091. IEEE. ↗
  8. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, pages 1505–1518. ↗
  9. Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244–250. IEEE. find
  10. Fanar, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus’ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, and Chaoyi Ruan. 2025. Fanar: An Arabic-Centric Multimodal Generative AI Platform. arxXiv:2501.13944.
  11. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, pages 3451–3460. ↗
  12. Ajinkya Kulkarni and Hanan Aldarmaki. 2023. Yet another model for Arabic dialect identification. In Proceedings of ArabicNLP 2023, pages 435– 440, Singapore (Hybrid). Association for Computational Linguistics.
  13. Yeonghyeon Lee, KANGWOOK JANG, Jahyun Goo, Youngmoon Jung, and Hoi-Rin Kim. 2022. Fithubert: Going thinner and deeper for knowledge distillation of speech self-supervised learning. In 23rd Annual Conference of the International Speech Communication Association, IN-TERSPEECH 2022, pages 3588–3592. ISCA. ↗
  14. Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, et al. 2022. Self-supervised speech representation learning: A review. IEEE Journal of Selected Topics in Signal Processing, 16(6):1179–1210. ↗
  15. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. ↗
  16. Yifan Peng, Yui Sudo, Shakeel Muhammad, and Shinji Watanabe. 2023. Dphubert: Joint distillation and pruning of self-supervised speech models. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, volume 2023, pages 62–66. find
  17. Jiatong Shi, Dan Berrebbi, William Chen, Ho-Lam Chung, En-Pei Hu, Wei Ping Huang, Xuankai Chang, Shang-Wen Li, Abdelrahman Mohamed, Hung yi Lee, and Shinji Watanabe. 2023. Ml-superb: Multilingual speech universal performance benchmark. find
  18. Edward Storey, Naomi Harte, and Peter Bell. 2024. Language bias in self-supervised learning for automatic speech recognition. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 37–42. IEEE. find
  19. Rui Wang, Qibing Bai, Junyi Ao, Long Zhou, Zhixiang Xiong, Zhihua Wei, Yu Zhang, Tom Ko, and Haizhou Li. 2022. Lighthubert: Lightweight and configurable speech representation learning with once-for-all hidden-unit bert. Proc. Interspeech 2022, pages 1686–1690. find
  20. Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung yi Lee. 2021. Superb: Speech processing universal performance benchmark. find
  21. Luca Zampierin, Ghouthi Boukli Hacene, Bac Nguyen, and Mirco Ravanelli. 2024. Skill: Similarity-aware knowledge distillation for speech self-supervised learning. arXiv preprint arXiv:2402.16830. 7. ↗
  22. Language Resource References Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech recognition challenge in the wild: Arabic MGB-3. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 316–322. IEEE.
  23. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. ↗
  24. Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Yujun Wang, Zhao You, and Zhiyong Yan. 2021. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. find
  25. Ali Hamid Meftah, Mustafa A. Qamhan, Yasser Seddiq, Yousef A. Alotaibi, and Sid Ahmed Selouani. 2021. King saud university emotions corpus: Construction, analysis, evaluation, and comparison. IEEE Access, 9:54201–54219. ↗
  26. Hamdy Mubarak, Amir Hussein, Shammur Absar Chowdhury, and Ahmed Ali. 2021. QASR: QCRI Aljazeera Speech Resource. A Large Scale Annotated Arabic Speech Corpus. In Proc. of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2274–2285, Online. Association for Computational Linguistics.
  27. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sukhadia-2026-harness-lightweight-distilled-arabic-speech/