Elamur Research⌕
☰
home / papers / 2024 / Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialect

Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialect

Paper · 2024 · Salima Mdhaffar, Haroun Elleuch, Fethi Bougares et al.

Summary

The paper asks which pretrained speech encoder works best when both the language and the labelled data are scarce. The test case is spoken Tunisian Arabic in a train-station dialogue domain. The authors fine-tune fourteen self-supervised encoders and two Whisper models on TARIC-SLU, using the same setup for ASR and SLU. The SLU side covers speech-act classification and slot filling. The encoders fall into four groups:

  • monolingual English or French models,
  • multilingual models,
  • bilingual SONAR variants,
  • encoders refined with a text-to-speech teacher-student step (SAMU-XLSR, SONAR).

The multilingual w2v-BERT 2.0 gives the best transcription and slot-filling scores. SAMU-XLSR, with its semantic refinement, comes close behind. Whisper transcribes competitively but extracts semantics less well than most SSL encoders. Error breakdowns by acoustic difficulty and by number of concepts show that model rankings shift across conditions. The study is among the first to test w2v-BERT 2.0 and SONAR for SLU. Code and models are released as a SpeechBrain recipe, which gives Arabic-dialect speech work a reproducible baseline.

Key points

  • Data: TARIC-SLU adds semantic labels to the TARIC corpus, recorded in Tunisian train stations. It has over 2,000 dialogues from 109 speakers. The training split holds 15,752 utterances, about 7.5 hours of audio.
  • Annotation scheme: 62 concept tags (such as departure city or ticket price) and 3 speech acts (directives-answer, directives-query, politeness).
  • Metrics: WER for ASR, SAER for speech acts, and two slot-filling scores. COER counts concept labels only, while CVER also requires the exact value.
  • Best model: w2v-BERT 2.0 (600M parameters, 4.5M hours, 143 languages) reaches about 21.5% test WER and about 46.2% test CVER.
  • Refinement and monolingual models:
    • SAMU-XLSR improves on its base XLS-R-128 for both ASR and SLU.
    • SONAR-ARB beats the English and French SONAR variants.
    • The English data2vec 2.0 has weak WER but competitive concept scores.

Interesting ideas

  • I1. Bucketing by upstream difficulty: test utterances are grouped by how badly the best ASR system transcribes them, then each downstream model is scored per group, which reveals robustness differences that a single average hides.
  • I2. Concept count as a complexity measure: the number of gold semantic tags per utterance serves as a cheap complexity proxy, and some models turn out strongest on the densest utterances.
  • I3. Label-only vs label-plus-value scoring: slot filling is scored twice, once for the label alone and once with the exact value, which separates "found the right slot" from "filled it correctly".
  • I4. Transcription accuracy is not understanding: a model with mediocre transcripts can still pick out concepts well, and Whisper shows the reverse, so each stage needs its own measurement.
  • I5. Flat inline semantic markup: the speech act comes as the first token, each concept span gets an opening tag, and one shared closing symbol ends every span, so the full semantic frame fits into one linear sequence.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Sentences could be grouped by how much the witness parsers disagree, with frame and answer accuracy reported per group. This would show which components hold up when the tree is shaky.
  • I2 (frames): Frame-extraction errors could be plotted against the number of predicates or roles per sentence. This would show whether the role labeller breaks down on dense clauses or on sparse ones.
  • I3 (frames): Semantic roles could be scored the same way, first on the VerbNet role label and then on the exact argument span. This would tell labelling mistakes apart from span or attachment mistakes.
  • I4 (learn): Frame and world-state quality should be evaluated directly instead of assuming LAS gains carry over downstream. Any parser change could then be judged by its effect on answers.
  • I5 (learn): A compact bracketed format like this could serve as the output schema for LLM teacher tiers producing silver frames. It is easy to parse back and to diff against the parser's own frames.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Laboratoire Informatique d'AvignonAvignon · France7 works · 436 citations🎓Université d'Avignon et des Pays de VaucluseAvignon · France6 works · 603 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksSpeech & spoken language · 480 worksLow-resource languages · 267 works

Cite

Cite as Salima Mdhaffar, Haroun Elleuch, Fethi Bougares and Yannick Estève. 2024. Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialect. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.12

@inproceedings{mdhaffar2024performance,
  title = {{Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialect}},
  author = {Salima Mdhaffar, Haroun Elleuch, Fethi Bougares and Yannick Estève},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.12},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.12}
}

References

26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021. Xls-r: Self-supervised cross-lingual speech representation learning at scale. pages arXiv–2111.
  2. Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. 2022a. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In International Conference on Machine Learning. ↗
  3. Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. 2022b. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pages 1298–1312. PMLR. ↗
  4. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems.
  5. Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, et al. 2023. Seamless: Multilingual expressive and streaming speech translation. arXiv preprint arXiv:2312.05187. ↗
  6. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, et al. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing. ↗
  7. Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. 2019. An Unsupervised Autoregressive Model for Speech Representation Learning. In Proc. Interspeech 2019. find
  8. Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In ASRU 2021. find
  9. Alexis Conneau, Ankur Bapna, Yu Zhang, Min Ma, Patrick von Platen, Anton Lozhkov, Colin Cherry, Ye Jia, Clara Rivera, Mihir Kale, et al. 2022. XTREME-S: Evaluating Cross-lingual Speech Representations. In Proc. Interspeech 2022, pages 3248– 3252. find
  10. Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023. Sentence-level multimodal and language-agnostic representations. arXiv preprint arXiv:2308.11466. ↗
  11. Solène Evain, Manh Ha Nguyen, Hang Le, Marcely Zanon Boito, Salima Mdhaffar, Sina Alisamir, Ziyi Tong, et al. 2021. Task agnostic and task specific self-supervised learning from speech with lebenchmark. In NeurIPS. find
  12. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891.
  13. S. Ghannay, A. Caubrière, Y. Estève, A. Laurent, and E. Morin. 2018. End-to-end named entity extraction from speech. CoRR, abs/1805.12045. ↗
  14. Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022. Bitext mining using distilled sentence representations for low-resource languages. arXiv preprint arXiv:2205.12654. ↗
  15. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM TASLP. ↗
  16. Sameer Khurana, Antoine Laurent, and James Glass. 2022. Samu-xlsr: Semantically-aligned multimodal utterance-level cross-lingual speech representation. Journal of Selected Topics in Signal Processing. ↗
  17. Gaëlle Laperrière, Valentin Pelloin, Antoine Caubrière, Salima Mdhaffar, Nathalie Camelin, Sahar Ghannay, Bassam Jabaian, and Yannick Estève. 2022. The spoken language understanding media benchmark dataset in the era of deep learning: data updates, training and evaluation tools. In LREC, pages 1595– 1602. find
  18. Andy T Liu, Shang-Wen Li, and Hung-yi Lee. 2021. Tera: Self-supervised learning of transformer encoder representation for speech. IEEE/ACM TASLP. ↗
  19. Abir Masmoudi, Yannick Estève, Mariem Ellouze Khmekhem, Fethi Bougares, and Lamia Hadrich Belguith. 2014. Phonetic tool for the tunisian arabic. In Spoken Language Technologies for Under-Resourced Languages. Citeseer. find
  20. Salima Mdhaffar, Fethi Bougars, Renato De mori, Salah Zaiem, Mirco Ravanelli, and Yannick Estève. 2024. Taric-slu: A tunisian dataset for spoken language understanding. LREC. find
  21. Salima Mdhaffar, Valentin Pelloin, Antoine Caubrière, Gaëlle Laperrière, Sahar Ghannay, Bassam Jabaian, Nathalie Camelin, and Yannick Estève. 2022. Impact analysis of the use of speech and language models pretrained by self-supersivion for spoken language understanding. In LREC 2022. find
  22. Hemant Yadav, Sreyan Ghosh, Yi Yu, and Rajiv Ratn Shah. 2020. End-to-end named entity recognition from english speech. Interspeech. find
  23. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038. ↗
  24. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2023. Scaling speech technology to 1,000+ languages. arXiv preprint arXiv:2305.13516. ↗
  25. Shu Wen Yang, Po Han Chi, Yung Sung Chuang, Cheng I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan Ting Lin, et al. 2021. Superb: Speech processing universal performance benchmark. In 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021, pages 3161–3165. International Speech Communication Association. ↗
  26. A Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mdhaffar-2024-performance-analysis-speech-encoders-low/