Elamur Research⌕
☰
home / papers / 2023 / Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition

Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition

Summary

Tunisian Arabic has no standard spelling, few annotated recordings, and speakers often mix in French and English. Large multilingual and Standard Arabic speech models do very badly on it. The authors build public resources and a baseline system to address this. They assembled text from the Tunisiya corpus and scraped code-switched text from the web. They recorded read speech through a crowd tool and hand-annotated radio and podcast audio that mixes three languages. They also gathered unlabeled TV audio.

Their recognizer fine-tunes a WavLM encoder with a CTC head and rescores outputs with a KenLM 4-gram model. Pseudo-labels on the unlabeled audio add self-training. For code-switched speech, a small mixer network combines frozen Tunisian, French and English recognizers.

The system clearly beats earlier Tunisian results, and a better language model gives the largest gains. A human evaluation suggests that automatic WER exaggerates errors because valid spellings vary. All data, models and code are released, giving the community a hard, realistic benchmark for three-language dialectal ASR.

Key points

  • Off-the-shelf models (MMS, Whisper Large v2, an Arabic wav2vec2.0) score roughly 74–139% WER on Tunisian test sets.
  • New datasets released on Zenodo and Hugging Face:
    • TunSwitch TO: read speech from 89 contributors, 2,631 phrases.
    • TunSwitch CS: about 8h of training audio, roughly 13.9% French and 13.3% English words, about five times the mixing found in STAC.
    • Unlabeled TV audio: 317h, reduced to 153h after cleaning.
  • The released text includes about 4.3M non-code-switched words and a much smaller code-switched set.
  • The English-only WavLM encoder worked better than the 53-language wav2vec2.0 XLSR.
  • Best WERs:
    • TARIC: 10.55, versus 22.6 in earlier work.
    • IWSLT: 38.86, versus 41.5.
    • TunSwitch TO: 25.54 with self-training and an out-of-domain LM, versus 41.45 with no LM.
  • On the code-switched audio, the human annotators agreed only 80.4% of the time.

Interesting ideas

  • I1. Annotators wrapped French and English words in inline language tags even though no model used them, keeping the data open to later code-switching work and usage statistics. — For Mova (tokens): Mark foreign or borrowed tokens (e.g., French phrases in English text) during annotation so the tagger and lemmatizer can treat them separately later.
  • I2. Rather than training one model on scarce mixed data, they froze three single-language experts and trained only a small BiLSTM "mixer" over their concatenated outputs on the little mixed data available. — For Mova (tree): A lightweight learned combiner over the frozen perceptron parser and gparse could merge their arc scores on hard or out-of-domain sentences without retraining either parser.
  • I3. When pseudo-labeled audio was added, retraining from scratch beat continuing to fine-tune the first model. — For Mova (learn): When silver trees from witness parsers are added, compare a fresh retrain against incremental fine-tuning instead of assuming continued training is better.
  • I4. They ran a human evaluation to separate true recognition errors from mismatches caused by inconsistent spelling in the reference transcripts. — For Mova (learn): Alongside exact-match metrics, have humans or a teacher judge whether disagreements with gold are real errors or acceptable variants (e.g., equally valid UD attachments), so that evaluation noise is not mistaken for model weakness.
  • I5. Sentences used as recording prompts were removed from the language-model corpus, preventing test leakage into the LM. — For Mova (learn): Systematically remove evaluation sentences (fables, school tests) from any corpus used to build compiled knowledge or LMs, as a built-in leakage control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

18 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 14 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.11327 · arXiv 2309.11327

@misc{abdallah2023leveraging,
  title = {{Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition}},
  author = {Abdallah, Ahmed Amine Ben and Kabboudi, Ata and Kanoun, Amir and Zaiem, Salah},
  year = {2023},
  doi = {10.48550/arxiv.2309.11327},
  eprint = {2309.11327},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.11327}
}

References

25 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022. find our card
  2. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli, “Scaling speech technology to 1,000+ languages,” arXiv, 2023. find our card
  3. Inès Zribi, Rahma Boujelbane, Abir Masmoudi, Mariem Ellouze, Lamia Hadrich Belguith, and Nizar Habash, “A conventional orthography for tunisian arabic.,” in LREC, 2014, pp. 2355–2361. find
  4. Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Ismail Babaoğlu, “Tunisian dialect sentiment analysis: a natural language processing-based approach,” Computación y Sistemas, vol. 22, no. 4, 2018. find
  5. Abir Masmoudi, Fethi Bougares, Mariem Ellouze, Yannick Estève, and Lamia Belguith, “Automatic speech recognition system for tunisian dialect,” Language Resources and Evaluation, vol. 52, pp. 249–267, 2018. find
  6. Abir Messaoudi, Hatem Haddad, Chayma Fourati, Moez Ben-Haj Hmida, Aymen Ben Elhaj Mabrouk, and Mohamed Graiet, “Tunisian dialectal end-to-end speech recognition based on deepspeech,” Procedia Computer Science, vol. 189, pp. 183– 190, 2021. find
  7. Jonatas Grosman, “Fine-tuned XLSR-53 large model for speech recognition in Arabic,” 2021. find
  8. Lotfi Sayahi, “Code-switching and language change in tunisia,” vol. 2011, no. 211, pp. 113–133, 2011. find
  9. Xian Shi, Qiangze Feng, and Lei Xie, “The asru 2019 mandarin-english code-switching speech recognition challenge: Open datasets, tracks, methods and results,” arXiv preprint arXiv:2007.05916, 2020. ↗
  10. Djegdjiga Amazouz, Martine Adda-Decker, and Lori Lamel, “Addressing Code-Switching in French/Algerian Arabic Speech,” in Interspeech 2017, Stockholm, Sweden, Aug. 2017, pp. 62–66. find
  11. Injy Hamed, Pavel Denisov, Chia-Yu Li, Mohamed Elmahdy, Slim Abdennadher, and Ngoc Thang Vu, “Investigations on speech recognition systems for low-resource dialectal arabic– english code-switching speech,” Computer Speech & Language, vol. 72, pp. 101278, 2022. find
  12. Sunayana Sitaram, Khyathi Raghavi Chandu, Sai Krishna Rallabandi, and Alan W Black, “A survey of code-switched speech and language processing,” 2019. find
  13. Karen McNeil, “Tunisian arabic corpus: Creating a written corpus of an ‘unwritten’language,” Arabic corpus linguistics, vol. 30, 2018. find
  14. Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang, “doccano: Text annotation tool for human,” 2018. find
  15. Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill, “Pyannote. audio: neural building blocks for speaker diarization,” in ICASSP. IEEE, 2020. find
  16. Marcely Zanon Boito, John Ortega, Hugo Riguidel, Antoine Laurent, Loı̈c Barrault, Fethi Bougares, Firas Chaabani, Ha Nguyen, Florentin Barbier, Souhir Gahbiche, et al., “On-trac consortium systems for the iwslt 2022 dialect and low-resource speech translation tasks,” . find
  17. Anastasopoulos Antonios, Barrault Loc, Luisa Bentivogli, Marcely Zanon Boito, Bojar Ondřej, Roldano Cattoni, Currey Anna, Dinu Georgiana, Duh Kevin, Elbayad Maha, et al., “Findings of the iwslt 2022 evaluation campaign.,” in IWSLT 2022. ACL, 2022. find our card
  18. Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid, and Mirco Ravanelli, “Speech self-supervised representation benchmarking: Are we doing it right?,” arXiv preprint arXiv:2306.00452, 2023. ↗
  19. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al., “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” 2021. find our card
  20. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE JSTSP, 2022. find
  21. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton, “Speech recognition with deep recurrent neural networks,” in ICASSP. IEEE, 2013, pp. 6645–6649. find
  22. Kenneth Heafield, “KenLM: Faster and smaller language model queries,” in Proceedings of the Sixth Workshop on Statistical Machine Translation, Edinburgh, Scotland, July 2011, pp. 187–197, ACL. find our card
  23. Dan Berrebbi, Ronan Collobert, Samy Bengio, Navdeep Jaitly, and Tatiana Likhomanenko, “Continuous pseudo-labeling from the start,” ICLR, 2023. find
  24. Brian Yan, Matthew Wiesner, Ondřej Klejch, Preethi Jyothi, and Shinji Watanabe, “Towards zero-shot code-switched speech recognition,” in ICASSP, 2023. find
  25. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber, “Common voice: A massively-multilingual speech corpus,” 2020. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abdallah-2023-leveraging-data-collection-unsupervised-learning/