Elamur Research⌕
☰
home / papers / 2023 / On the Robustness of Arabic Speech Dialect Identification

On the Robustness of Arabic Speech Dialect Identification

Summary

Arabic dialect identification (ADI) systems often act as filters in large web-scale pipelines that gather speech for training ASR. They are then applied to audio that differs from what they were trained on. The authors ask how badly such domain shift hurts them. They build ADI classifiers from self-supervised speech models (XLS-R 300m and HuBERT-large-ll60k) in two ways. The first is end-to-end fine-tuning with a pooling head. The second is a lightweight classifier over length-normalized counts of HuBERT k-means pseudo-units, offered as an alternative to i/x-vectors. Training uses ADI-17, ADI-5 and MGB-2. Two new YouTube collections serve as realistic test beds: YouTube City, gathered with city-name keywords, and YouTube Dramas, drawn from TV series. They also try self-training on silver-labelled YouTube City audio and run a human study on a sample of the data to see what drives model decisions. Domain shift turns out to be a serious problem. Self-training reduces the damage but probably does not remove it under real collection conditions. This matters for anyone who uses ADI to bootstrap dialectal speech corpora.

Key points

  • Data: ADI-5 (~74 h, 5 classes including MSA) and ADI-17 (~3,000 h, 17 dialects, no MSA). The new ADI-17 + MSA set adds MGB-2 and the MSA part of ADI-5. YouTube Dramas (24.8 h) covers 7 dialects, with three series per dialect.
  • YouTube City pipeline: the source pool is 238.2K videos (~62k hours) found via keywords for 319 cities in 22 countries. A sample of 91.4K videos is processed in four steps:
    • pyannote.audio VAD splits the audio and keeps segments over 3 s, giving ~2M segments.
    • lang-id-voxlingua107-ecapa marks ~1.7M of these as Arabic.
    • HuBERT-17MSA assigns silver dialect labels.
  • Only 442.6K utterances (26.58%) received a predicted dialect that matched the channel's country of origin.
  • Fine-tuned models are xls-r-300m-17, HuBERT-17 and HuBERT-17MSA. Each was tuned with a random search of 30 runs on a fixed budget. The search covered:
    • freeze steps and thaw depth;
    • LNA fine-tuning;
    • a tri-state learning-rate schedule (10% warm-up, 50% plateau, 40% decay).
  • Pseudo-unit features come from HuBERT Base layer 10, with k ∈ {200…1000} for k-means. The best setting was 1000 clusters, batch 256 and learning rate 1e-2.

Interesting ideas

  • I1. Silver data is not treated as one pool: it is split into a metadata-agreeing ("surrogate") group plus high, medium and low confidence tiers, so each tier's value for self-training can be measured separately. — For Mova (learn): split silver UD parses into tiers by witness-parser agreement and confidence, then measure which tiers help or hurt the parser rather than ingesting all of them.
  • I2. A deliberately crude baseline (length-normalized histograms of discrete units fed to a single layer) is used because it is less able to latch onto channel artefacts, which exposes how much a stronger model relies on non-linguistic cues. — For Mova (learn): keep an order-free unit-count classifier as a negative control, so a structured tree/frame reader must clearly beat it to show it is really using structure.
  • I3. The authors tune with random search under a fixed compute budget, on the reasoning that only a few hyperparameters really matter, instead of running an exhaustive grid. — For Mova (tree): tune gparse (thaw depth, freeze steps, learning rate) with budgeted random search and record which knobs actually move LAS.
  • I4. Adaptation is partial: the head is trained first with the backbone frozen, and then only selected layers are opened up, or only layer norm and attention as in LNA. — For Mova (tree): when adapting gparse to a new treebank or UD dialect conversion, unfreeze only norm/attention parameters to limit forgetting of the original treebank.
  • I5. The evaluation uses freshly collected, out-of-domain material together with a human look at a sample, rather than trusting held-out splits of the training corpus. — For Mova (answer): test readers on new fables or school texts from unseen sources and hand-inspect a sample of failures to separate genre shift from reasoning bugs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

12 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 10 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.03789 · arXiv 2306.03789

@misc{sullivan2023robustness,
  title = {{On the Robustness of Arabic Speech Dialect Identification}},
  author = {Sullivan, Peter and Elmadany, AbdelRahim and Abdul-Mageed, Muhammad},
  year = {2023},
  doi = {10.48550/arxiv.2306.03789},
  eprint = {2306.03789},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.03789}
}

References

34 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. A. Chowdhury, Y. Samih, M. Eldesouki, and A. Ali, “Effects of dialectal code-switching on speech modules: A study using Egyptian Arabic broadcast speech.” in INTERSPEECH, 2020, pp. 2382–2386. find
  2. R. Duroselle, D. Jouvet, and I. Illina, “Unsupervised regularization of the embedding extractor for robust language identification,” in Odyssey 2020-The Speaker and Language Recognition Workshop, 2020. find
  3. S. Nercessian, P. Torres-Carrasquillo, and G. Martinez-Montes, “Approaches for language identification in mismatched environments,” in 2016 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2016, pp. 335–340. find
  4. W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021. find our card
  5. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in Neural Information Processing Systems, vol. 33, pp. 12 449–12 460, 2020. find our card
  6. S.-w. Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin et al., “Superb: Speech processing universal performance benchmark,” arXiv preprint arXiv:2105.01051, 2021. ↗
  7. E. Singer, P. Torres-Carrasquillo, D. A. Reynolds, A. McCree, F. Richardson, N. Dehak, and D. Sturim, “The MITLL NIST LRE 2011 language recognition system,” in Odyssey 2012-The Speaker and Language Recognition Workshop. Citeseer, 2012. find
  8. A. Abad, “The L2F language recognition system for NIST LRE 2011,” in The NIST Language Recognition Evaluation Workshop, 2011. find
  9. A. Ali, S. Vogel, and S. Renals, “Speech recognition challenge in the wild: Arabic MGB-3,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 316–322. find our card
  10. A. Ali, S. Shon, Y. Samih, H. Mubarak, A. Abdelali, J. Glass, S. Renals, and K. Choukri, “The MGB-5 challenge: Recognition and dialect identification of dialectal Arabic speech,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 1026–1033. find
  11. S. O. Sadjadi, T. Kheyrkhah, A. Tong, C. S. Greenberg, D. A. Reynolds, E. Singer, L. P. Mason, J. Hernandez-Cordero et al., “The 2017 NIST language recognition evaluation.” in Odyssey, 2018, pp. 82–89. find
  12. S. A. Chowdhury, A. M. Ali, S. Shon, and J. R. Glass, “What does an end-to-end dialect identification model learn about non-dialectal information?” in INTERSPEECH, 2020, pp. 462–466. find
  13. W. Lin, M. Madhavi, R. K. Das, and H. Li, “Transformer-based arabic dialect identification,” in 2020 International Conference on Asian Language Processing (IALP). IEEE, 2020, pp. 192–196. find
  14. A. F. Martin, C. S. Greenberg, J. M. Howard, D. Bansé, G. R. Doddington, J. Hernández-Cordero, and L. P. Mason, “NIST language recognition evaluation—plans for 2015,” in Sixteenth Annual Conference of the International Speech Communication Association, 2015. find
  15. K. A. Lee, H. Li, L. Deng, V. Hautamäki, W. Rao, X. Xiao, A. Larcher, H. Sun, T. Nguyen, G. Wang et al., “The 2015 NIST language recognition evaluation: the shared view of i2r, fantastic4 and singams,” in INTERSPEECH 2016, vol. 2016, 2016, pp. 3211–3215. find
  16. O. Plchot, P. Matejka, O. Glembek, R. Fer, O. Novotnỳ, J. Pesan, L. Burget, N. Brummer, and S. Cumani, “BAT system description for NIST LRE 2015.” in Odyssey, 2016, pp. 166–173. find
  17. C. Yu, C. Zhang, S. Ranjan, Q. Zhang, A. Misra, F. Kelly, and J. H. Hansen, “UTD-CRSS system for the NIST 2015 language recognition i-vector machine learning challenge,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016, pp. 5835–5839. find
  18. D. Snyder, D. Garcia-Romero, A. McCree, G. Sell, D. Povey, and S. Khudanpur, “Spoken language recognition using x-vectors.” in Odyssey, 2018, pp. 105–111. find
  19. S. Shon, A. Ali, and J. Glass, “MIT-QCRI Arabic dialect identification system for the 2017 multi-genre broadcast challenge,” in 2017 ieee automatic speech recognition and understanding workshop (asru). IEEE, 2017, pp. 374–380. find
  20. A. E. Bulut, Q. Zhang, C. Zhang, F. Bahmaninezhad, and J. H. Hansen, “UTD-CRSS submission for MGB-3 Arabic dialect identification: Front-end and back-end advancements on broadcast speech,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 360–367. find
  21. W. Cai, Z. Cai, W. Liu, X. Wang, and M. Li, “Insights in-to-end learning scheme for language identification,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5209–5213. find
  22. Y.-A. Chung and J. Glass, “Improved speech representations with multi-target autoregressive predictive coding,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 2353–2358. find
  23. A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino et al., “XLS-r: Self-supervised cross-lingual speech representation learning at scale,” arXiv preprint arXiv:2111.09296, 2021. ↗ our card
  24. S. Shon, A. Ali, Y. Samih, H. Mubarak, and J. Glass, “ADI17: A fine-grained Arabic dialect identification dataset,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 8244–8248. find our card
  25. A. Ali, P. Bell, J. Glass, Y. Messaoui, H. Mubarak, S. Renals, and Y. Zhang, “The MGB-2 challenge: Arabic multi-dialect broadcast media recognition,” in 2016 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2016, pp. 279–284. find our card
  26. M. Abdul-Mageed, C. Zhang, A. Elmadany, and L. Ungar, “Toward micro-dialect identification in diaglossic and code-switched environments,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics, Nov. 2020, pp. 5855–5876. [Online]. Available: https://aclanthology.org/2020.emnlp-main.472 ↗
  27. H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M.-P. Gill, “pyannote.audio: neural building blocks for speaker diarization,” in ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing, Barcelona, Spain, May 2020. find
  28. H. Bredin and A. Laurent, “End-to-end speaker segmentation for overlap-aware resegmentation,” in Proc. Interspeech 2021, Brno, Czech Republic, August 2021. find
  29. M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, “SpeechBrain: A general-purpose speech toolkit,” 2021, arXiv:2106.04624. ↗
  30. J. Valk and T. Alumäe, “VoxLingua107: a dataset for spoken language recognition,” in Proc. IEEE SLT Workshop, 2021. find
  31. W. Cai, J. Chen, and M. Li, “Exploring the encoding layer and loss function in end-to-end speaker and language recognition system,” in Proc. Odyssey 2018 The Speaker and Language Recognition Workshop, 2018, pp. 74–81. find
  32. J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization.” Journal of machine learning research, vol. 13, no. 2, 2012. find
  33. X. Li, C. Wang, Y. Tang, C. Tran, Y. Tang, J. Pino, A. Baevski, A. Conneau, and M. Auli, “Multilingual speech translation from efficient finetuning of pretrained models,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 827– 838. find
  34. H. Bořil, A. Sangwan, and J. H. Hansen, “Arabic dialect identification-” is the secret in the silence?” and other observations,” in Thirteenth Annual Conference of the International Speech Communication Association, 2012. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sullivan-2023-robustness-arabic-speech-dialect-identification/