Elamur Research⌕
☰
home / papers / 2026 / Ara-Best-RQ: Multi Dialectal Arabic SSL

Ara-Best-RQ: Multi Dialectal Arabic SSL

Summary

Arabic speech is thinly covered by self-supervised speech models. Multilingual encoders are dominated by English and other high-resource languages, and the Arabic they do contain is mostly Modern Standard Arabic. Dialects, which differ strongly in sound and vocabulary, get little attention, and no large open dialectal audio collection existed for pre-training.

The authors gathered 5,640 hours of Creative Commons YouTube audio spanning 20 Arabic dialects. They also merged it with many public corpora into a deduplicated pool of 13,723 hours. On this data they pre-trained conformer encoders with the BEST-RQ objective at 300M and 600M parameters, using a streaming setup. They then fine-tuned the encoders for dialect identification (DID) and ASR.

The 300M model trained only on crawled data set a new state of the art in DID with under half the parameters of a Whisper-based competitor. On ASR it matched strong baselines such as w2v-BERT 2.0. The broader lesson is that pre-training targeted at one language family pays off compared with generic multilingual models. Models, code and data are promised as open releases.

Key points

  • Crawled corpus. About 35k CC video links from roughly 8,800 channels were screened for offensive content, leaving 26k videos. Silero VAD segmentation merged gaps under 250 ms, split segments over 20 s and dropped those under 1 s. The result is 3.86M segments totalling 5,640 h.
  • Combined corpus. The 13,723 h pool covers MSA, dialects and Classical Arabic (ClArTTS), drawing on sources such as ADI-17, MGB-2, QASR, MASC and SADA. It also includes CommonVoice 16.1 French, English and Italian, with English capped at 500 h and French at 396 h, both sampled for gender balance.
  • Architecture and training. Both models have 24 conformer layers and 8 heads, with widths of 848 and 1024. Training used Dynamic Chunk Training, 60% total masking, and a random-projection quantizer with 4,096 codes. The implementation is SpeechBrain.
  • Pre-training outcomes. The 300M model on combined data did not converge and was dropped. The 600M model on combined data reached the lowest validation loss (3.40), while the 600M model on crawled data alone showed overfitting.
  • ASR results (average WER over CV 19.0, MGB-3, MGB-5 and TARIC-SLU). Ara-BEST-RQ 300M scored 31.92, against 43.62 for HuBERT-large and 44.33 for XLS-R-128. The 600M combined model scored 30.76, close to w2v-BERT 2.0 at 30.34.

Interesting ideas

  • I1. Family-targeted pre-training. Pre-training on one language family, rather than a broad multilingual mix, gave a smaller model that beat larger generic ones, so data relevance can outweigh scale. — For Mova (tree): Pre-training or adapting gparse on English text close to Mova's target genres (school texts, fables) may beat generic web-scale encoders of the same size.
  • I2. Distrust noisy metadata. The authors ignored YouTube geotags as unreliable and instead labeled missing dialects with their own strongest DID classifier. — For Mova (learn): Treat source metadata (genre, register tags) as suspect, and derive silver labels from the best agreed-upon internal model rather than from header fields.
  • I3. Gate on convergence. The small model that failed to converge on more diverse data was explicitly excluded from evaluation, showing that capacity has to match data variety. — For Mova (learn): Add a convergence/capacity check before any trained parser or tagger enters the witness pool, so diverged models cannot pollute silver agreement.
  • I4. Random context limits. Chunk sizes and left context were randomly varied during training so the encoder copes with both short and long contexts. — For Mova (frames): Train coreference and frame resolution with randomly truncated preceding context, so they degrade gracefully when only a sentence or two of history is available.
  • I5. Cap and balance sources. Over-represented sources were capped, samples were balanced (by gender), and overlapping content between corpora was removed before merging. — For Mova (learn): Cap dominant genres, balance sources in training mixes, and deduplicate across corpora so evaluation fables or tests never leak into training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2603.21900 · arXiv 2603.21900

@misc{elleuch2026ara,
  title = {{Ara-Best-RQ: Multi Dialectal Arabic SSL}},
  author = {Elleuch, Haroun and Whetten, Ryan and Mdhaffar, Salima and Esteve, Yannick and Bougares, Fethi},
  year = {2026},
  doi = {10.48550/arxiv.2603.21900},
  eprint = {2603.21900},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.21900}
}

References

39 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexei Baevski et al., “wav2vec 2.0: A framework for self-supervised learning of speech representations,” NeurIPS, vol. 33, pp. 12449–12460, 2020. find our card
  2. Chung-Cheng Chiu et al., “Self-supervised learning with random-projection quantizer for speech recognition,” in ICML. PMLR, 2022, pp. 3915–3924. find
  3. Arun Babu et al., “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” arXiv preprint arXiv:2111.09296, 2021. ↗ our card
  4. Barrault Loı̈c et al., “Seamless: Multilingual expressive and streaming speech translation,” arXiv preprint arXiv: 2312.05187, 2023. ↗ our card
  5. Yu Zhang et al., “Google usm: Scaling automatic speech recognition beyond 100 languages,” arXiv preprint arXiv:2303.01037, 2023. ↗
  6. Amirbek Djanibekov et al., “Dialectal coverage and generalization in Arabic speech recognition,” in ACL, 2025, pp. 29490–29502. find
  7. Wei-Ning Hsu et al., “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM, vol. 29, pp. 3451–3460, 2021. find our card
  8. Solène Evain et al., “Lebenchmark: A reproducible framework for assessing self-supervised representation learning from speech,” arXiv preprint arXiv:2104.11462, 2021. ↗
  9. Phuong-Hang Le et al., “Pantagruel: Unified self-supervised encoders for french text and speech,” arXiv preprint arXiv:2601.05911, 2026. ↗
  10. Jesujoba O Alabi et al., “Afrihubert: A self-supervised speech representation model for african languages,” arXiv preprint arXiv:2409.20201, 2024. ↗
  11. Salima Mdhaffar et al., “Performance analysis of speech encoders for low-resource slu and asr in tunisian dialect,” in ArabicNLP, 2024, pp. 130–139. find
  12. Hawau Toyin et al., “ArTST: Arabic text and speech transformer,” in Proceedings of ArabicNLP 2023, Singapore (Hybrid), pp. 41–51, ACL. find
  13. Junyi Ao et al., “Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing,” arXiv preprint arXiv:2110.07205, 2021. ↗
  14. Lamya Alkanhal et al., “Aswat: Arabic audio dataset for automatic speech recognition using speech-representation learning,” in Proceedings of ArabicNLP 2023, Singapore (Hybrid), Dec. 2023, pp. 120–127, ACL. find
  15. Alexei Baevski et al., “Data2vec: A general framework for self-supervised learning in speech, vision and language,” in International conference on machine learning. PMLR, 2022. find
  16. Ryan Whetten et al., “Open implementation and study of best-rq for speech processing,” in ICASSPW). IEEE, 2024. find
  17. Silero Team, “Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier,” 2024. find
  18. Ajinkya Kulkarni et al., “Clartts: An open-source classical arabic text-to-speech corpus,” arXiv preprint arXiv:2303.00069, 2023. ↗ our card
  19. Rosana Ardila et al., “Common voice: A massively-multilingual speech corpus,” in LREC, Marseille, France, May 2020, pp. 4218–4222, ELRA. find our card
  20. Suwon Shon et al., “Adi17: A fine-grained arabic dialect identification dataset,” in ICASSP 2020, 2020, pp. 8244–8248. find our card
  21. Ahmed M. Ali et al., “Speech recognition challenge in the wild: Arabic mgb-3,” ASRU 2017, pp. 316–322, 2017. find our card
  22. Simone Bianco et al., “Arabceleb: Speaker recognition in arabic,” in AIxIA 2021 – Advances in Artificial Intelligence, Cham, 2022, pp. 338–347, Springer International Publishing. find
  23. Abdulkader Ghandoura et al., “Building and benchmarking an arabic speech commands dataset for small-footprint keyword spotting,” Engineering Applications of Artificial Intelligence, vol. 102, pp. 104267, 2021. find
  24. Nawar Halabi, Arabic Speech Corpus, Ph.D. thesis, University of Oxford, 2016. find
  25. Hedi Naouara et al., “Linto audio and textual datasets to train and evaluate automatic speech recognition in tunisian arabic dialect,” October 2024, Good Data Workshop, AAAI 2025. find our card
  26. Mohammad Al-Fetyani et al., “Masc: Massive arabic speech corpus,” in SLT 2022, 2023, pp. 1006–1013. find
  27. Rostislav Kolobov et al., “Mediaspeech: Multilanguage asr benchmark and dataset,” 2021. find
  28. Ahmed Ali et al., “The mgb-2 challenge: Arabic multi-dialect broadcast media recognition,” in SLT 2016, 2016, pp. 279–284. find our card
  29. Ahmed Ali et al., “Speech recognition challenge in the wild: Arabic mgb-3,” in ASRU 2017, 2017, pp. 316–322. find our card
  30. Ahmed Ali et al., “The mgb-5 challenge: Recognition and dialect identification of dialectal arabic speech,” in ASRU, 2019, pp. 1026–1033. find
  31. Maryam Khalifa Al Ali and Hanan Aldarmaki, “Mixat: A data set of bilingual emirati-English speech,” in LREC-COLING, Torino, Italia, May 2024, pp. 222–226, ELRA and ICCL. find
  32. Mohammad M. Khader et al., “Munazarat 1.0: A corpus of Arabic competitive debates,” in OSACT - LREC-COLING 2024. May 2024, ELRA and ICCL. find our card
  33. Hamdy Mubarak et al., “QASR: QCRI aljazeera speech resource a large scale annotated Arabic speech corpus,” in ACL, Aug. 2021, pp. 2274–2285. find
  34. Sadeen Alharbi et al., “Sada: Saudi audio dataset for arabic,” in ICASSP 2024, 2024, pp. 10286–10290. find
  35. Abdulrahman Alalshekmubarak and Leslie S. Smith, “On improving the classification capability of reservoir computing for arabic speech recognition,” in ICANN 2014, Cham, 2014, pp. 225–232, Springer International Publishing. find
  36. Ahmed Abdallah et al., “Leveraging data collection and unsupervised learning for code-switched tunisian arabic automatic speech recognition,” in ICASSP 2024, 2024, pp. 12607–12611. find our card
  37. Mirco Ravanelli et al., “Open-source conversational ai with speechbrain 1.0,” Journal of Machine Learning Research, vol. 25, no. 333, pp. 1–11, 2024. find
  38. Salima Mdhaffar et al., “Taric-slu: A tunisian benchmark dataset for spoken language understanding,” in LREC-COLING 2024, 2024, pp. 15606–15616. find
  39. Haroun Elleuch et al., “Adi-20: Arabic dialect identification dataset and models,” in Proceedings of Interspeech, 2025. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elleuch-2026-ara-best-rq-multi-dialectal/