Elamur Research⌕
☰
home / papers / 2024 / UM IWSLT 2024 Low-Resource Speech Translation: Combining Maltese and North Levantine Arabic

UM IWSLT 2024 Low-Resource Speech Translation: Combining Maltese and North Levantine Arabic

Summary

The paper describes the University of Malta's submission to the IWSLT 2024 low-resource track. The task was speech translation into English from Maltese and from North Levantine Arabic (APC), two Semitic languages with related histories. The team used an unconstrained cascade. For speech recognition, a Wav2Vec 2.0 XLS-R model was fine-tuned, and its transcripts were corrected with n-gram language models. For translation, NLLB was fine-tuned in two stages: first on a larger external corpus, then on in-domain or dialect-specific data.

The authors also converted Maltese text into Arabic script, so that Maltese parallel data could be added to the training of the Arabic translation system. The Maltese pipeline reached 52.4 BLEU on the test set. The Arabic pipeline stayed near 5 BLEU because no North Levantine speech data was available and the recogniser was trained on MSA Common Voice audio.

The work matters because it shows two things. Staged fine-tuning works well for under-resourced languages. Transliteration can share data between related languages that use different scripts.

Key points

  • Maltese ASR data: about 50 hours from Common Voice 7.0 and MASRI.
  • Arabic ASR data: a 50-hour MSA subset of Common Voice. The authors found no APC speech corpus, and they skipped Tunisian data because APC is closer to MSA.
  • Maltese MT: the best system (MLT LAT) was fine-tuned first on OPUS-100 and then on CV+MASRI. Fine-tuning on OPUS-100 alone was worse than the baseline.
  • Maltese pipeline results: test BLEU 52.4 with 3-gram and 4-gram LM correction, and slightly lower with a 6-gram LM. Test WER was about 0.094 on CV and 0.233 on MASRI.
  • Arabic results: the best MT recipe was MSA first, then APC (plus AEB).
    • Adding transliterated Maltese (MLT ARA) changed Arabic dev BLEU by only 0.2.
    • The same system reached 37.0 BLEU on the transliterated Maltese dev set.
    • The best Arabic test scores were BLEU 5.09, COMET 53.78 and ChrF 24.50 (Contrastive 1).
    • The previous year's Maltese submission had reached only 0.7 BLEU.

Interesting ideas

  • I1. Two-stage fine-tuning. Train first on a large, loosely matched corpus, then on a small, well-matched one; jumping straight to the general data alone actually hurt. This ordering reliably beats either source used by itself. — For Mova (learn): Pretrain gparse on broad silver treebanks, then fine-tune on the small gold set of the target genre, and test each stage on its own.
  • I2. Transliteration as a bridge. Rewriting a related language into the target script lets its labelled data join training. Earlier work cited in the paper refines this by choosing between transliteration and translation according to word etymology. — For Mova (tree): When converting treebanks from other UD dialects, map only the constructions that truly correspond and leave the rest untouched, so the convertible part of foreign data can be reused safely.
  • I3. Two-sided evaluation of auxiliary data. When adding helper data, measure both the gain on the new domain and the cost on the original one; here the cost was just 0.2 BLEU while Maltese improved greatly. — For Mova (learn): Any added silver or cross-domain corpus should be accepted only after checking regressions on the original benchmark, which acts as a control.
  • I4. Cheap n-gram post-correction. Small n-gram language models were used to fix recogniser output, and several orders were compared; the lower ones were enough. — For Mova (tokens): A compact, inspectable n-gram model could rerank ambiguous tagger or lemma candidates, and the best order can be chosen empirically.
  • I5. Per-stage scoring in a cascade. Reporting WER separately from BLEU showed that the weak Arabic result came from the speech recogniser rather than the translator. — For Mova (answer): Log a score for every pipeline step behind each answer, so that wrong answers can be traced to the step that caused them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Speech & spoken language · Low-resource languages · Semitic languages

Institutions

🎓University of MaltaMsida · Malta12 works · 573 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.iwslt-1.14

@inproceedings{nabhani2024um,
  title = {{UM IWSLT 2024 Low-Resource Speech Translation: Combining Maltese and North Levantine Arabic}},
  author = {Nabhani, Sara and Williams, Aiden and Jannat, Miftahul and Belcher, Kate Rebecca and Galea, Melanie and Taylor, Anna and Micallef, Kurt and Borg, Claudia},
  booktitle = {Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.iwslt-1.14},
  url = {https://doi.org/10.18653/v1/2024.iwslt-1.14}
}

References

37 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Milind Agarwal, Sweta Agrawal, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Mingda Chen, William Chen, Khalid Choukri, Alexandra Chronopoulou, Anna Currey, Thierry Declerck, Qianqian Dong, Kevin Duh, Yannick Estève, Marcello Federico, Souhir Gahbiche, Barry Haddow, Benjamin Hsu, Phu Mon Htut, Hirofumi Inaguma, Dávid Javorský, John Judge, Yasumasa Kano, Tom Ko, Rishu Kumar, Pengwei Li, Xutai Ma, Prashant Mathur, Evgeny Matusov, Paul McNamee, John P. McCrae, Kenton Murray, Maria Nadejde, Satoshi Nakamura, Matteo Negri, Ha Nguyen, Jan Niehues, Xing Niu, Atul Kr. Ojha, John E. Ortega, Proyag Pal, Juan Pino, Lonneke van der Plas, Peter Polák, Elijah Rippeth, Elizabeth Salesky, Jiatong Shi, Matthias Sperber, Sebastian Stüker, Katsuhito Sudoh, Yun Tang, Brian Thompson, Kevin Tran, Marco Turchi, Alex Waibel, Mingxuan Wang, Shinji Watanabe, and Rodolfo Zevallos. 2023. find
  2. FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN. In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 1–61, Toronto, Canada (in-person and online). Association for Computational Linguistics. find our card
  3. Diego Alves, Askars Salimbajevs, and Mārcis Pinnis. 2020. Data Augmentation for Pipeline-Based Speech Translation. find
  4. Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, Jie Chen, Jingdong Chen, Zhijie Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Ke Ding, Niandong Du, Erich Elsen, Jesse Engel, Weiwei Fang, Linxi Fan, Christopher Fougner, Liang Gao, Caixia Gong, Awni Hannun, Tony Han, Lappi Vaino Johannes, Bing Jiang, Cai Ju, Billy Jun, Patrick LeGresley, Libby Lin, Junjie Liu, Yang Liu, Weigao Li, Xiangang Li, Dongpeng Ma, Sharan Narang, Andrew Ng, Sherjil Ozair, Yiping Peng, Ryan Prenger, Sheng Qian, Zongfeng Quan, Jonathan Raiman, Vinay Rao, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Kavya Srinet, Anuroop Sriram, Haiyuan Tang, Liliang Tang, Chong Wang, Jidong Wang, Kaifu Wang, Yi Wang, Zhijian Wang, Zhiqian Wang, Shuang Wu, Likai Wei, Bo Xiao, Wen Xie, Yan Xie, Dani Yogatama, Bin Yuan, Jun Zhan, and Zhenyao Zhu. 2016. Deep speech 2: end-to-end speech recognition in english and mandarin. In Proceedings of the 33rd International Conference on International Conference on Machine Learning -Volume 48, ICML’16, page 173–182. find our card
  5. JMLR.org. Antonios Anastasopoulos, Loïc Barrault, Luisa Bentivogli, Marcely Zanon Boito, Ondřej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad, Clara Emmanuel, Yannick Estève, Marcello Federico, Christian Federmann, Souhir Gahbiche, Hongyu Gong, Roman Grundkiewicz, Barry Haddow, Benjamin Hsu, Dávid Javorský, Vĕra Kloudová, Surafel Lakew, Xutai Ma, Prashant Mathur, Paul McNamee, Kenton Murray, Maria Nǎdejde, Satoshi Nakamura, Matteo Negri, Jan Niehues, Xing Niu, John Ortega, Juan Pino, Elizabeth Salesky, Jiatong Shi, Matthias Sperber, Sebastian Stüker, Katsuhito Sudoh, Marco Turchi, Yogesh Virkar, Alexander Waibel, Changhan Wang, and Shinji Watanabe. 2022. Findings of the IWSLT 2022 evaluation campaign. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 98–157, Dublin, Ireland (in-person and online). Association for Computational Linguistics. find our card
  6. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4218–4222, Marseille, France. European Language Resources Association. find our card
  7. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2022. Xls-r: Self-supervised cross-lingual speech representation learning at scale. find our card
  8. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, volume 33, pages 12449–12460. Curran Associates, Inc. find our card
  9. Albert Borg and Marie Azzopardi-Alexander. 1997. Maltese: Descriptive Grammars. Routledge, London and New York. find
  10. Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2020. Unsupervised cross-lingual representation learning for speech recognition. CoRR, abs/2006.13979. ↗
  11. John E. Ortega, Rodolfo Zevallos, and William Chen. 2023. QUESPA submission for the IWSLT 2023 dialect and low-resource speech translation tasks. In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 261–268, Toronto, Canada (in-person and online). Association for Computational Linguistics. find
  12. Mark JF Gales, Kate M Knill, Anton Ragni, and Shakti P Rath. 2014. Speech recognition and keyword spotting for low-resource languages: Babel project research at CUED. In Fourth International workshop on spoken language technologies for under-resourced languages (SLTU-2014), pages 16–23. International Speech Communication Association (ISCA). find
  13. Albert Gatt and Slavomír Čéplö. 2013. Digital Corpora and Other Electronic Resources for Maltese. In Proceedings of the International Conference on Corpus Linguistics, pages 96–97. UCREL, Lancaster, UK. find
  14. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala– English. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6098–6111, Hong Kong, China. Association for Computational Linguistics. find our card
  15. Nizar Habash, Nasser Zalmout, Dima Taji, Hieu Hoang, and Maverick Alzate. 2017. A parallel corpus for evaluating machine translation between Arabic and European languages. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 235–241, Valencia, Spain. Association for Computational Linguistics. find
  16. Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland. Association for Computational Linguistics. find our card
  17. Carlos Daniel Hernandez Mena, Albert Gatt, Andrea DeMarco, Claudia Borg, Lonneke van der Plas, Amanda Muscat, and Ian Padovani. 2020. MASRI-HEADSET: A Maltese corpus for speech recognition. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6381–6388, Marseille, France. European Language Resources Association. find our card
  18. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics. corpus and BERT models for Maltese. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 90–101, Hybrid. Association for Computational Linguistics. find our card
  19. Kurt Micallef, Nizar Habash, Claudia Borg, Fadhl Eryani, and Houda Bouamor. 2024. Cross-lingual transfer from related languages: Treating low-resource Maltese as multilingual code-switching. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1014–1025, St. Julian’s, Malta. Association for Computational Linguistics. find
  20. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672. ↗
  21. Daniel Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Cubuk, and Quoc Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. find our card
  22. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. MLS: A large-scale multilingual dataset for speech research. In Interspeech 2020. ISCA. find our card
  23. Kathrein Abu Kwaik, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnik. 2018. A lexical distance study of Arabic dialects. Procedia Computer Science, 142:2–13. Arabic Computational Linguistics. find
  24. Alexandre Magueresse, Vincent Carles, and Evan Heetderks. 2020. Low-resource languages: A review of past work and future challenges. Preprint, arXiv:2006.07264. ↗
  25. Kurt Micallef, Fadhl Eryani, Nizar Habash, Houda Bouamor, and Claudia Borg. 2023. Exploring the impact of transliteration on NLP performance: Treating Maltese as an Arabic dialect. In Proceedings of the Workshop on Computation and Written Language (CAWL 2023), pages 22–32, Toronto, Canada. Association for Computational Linguistics. find
  26. Kurt Micallef, Albert Gatt, Marc Tanti, Lonneke van der Plas, and Claudia Borg. 2022. Pre-training data quality and quantity for a low-resource language: New Hashem Sellat, Shadi Saleh, Mateusz Krubiński, Adam Pospíšil, Petr Zemánek, and Pavel Pecina. 2023. UFAL parallel corpus of north levantine 1.0. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  27. Nivedita Sethiya and Chandresh Kumar Maurya. 2023. End-to-end speech-to-text translation: A survey. Preprint, arXiv:2312.01053. ↗
  28. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. CoRR, abs/2008.00401. ↗
  29. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). find
  30. Abdul Waheed, Bashar Talafha, Peter Sullivan, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2023. Voxarabica: A robust dialect-aware arabic speech recognition system. Preprint, arXiv:2310.11069. ↗
  31. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 993–1003, Online. Association for Computational Linguistics. find our card
  32. Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020. Fairseq S2T: Fast speech-to-text modeling with fairseq. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: System Demonstrations, pages 33–39, Suzhou, China. Association for Computational Linguistics. find our card
  33. Aiden Williams, Kurt Abela, Rishu Kumar, Martin Bär, Hannah Billinghurst, Kurt Micallef, Ahnaf Mozib Samin, Andrea DeMarco, Lonneke van der Plas, and Claudia Borg. 2023a. UM-DFKI Maltese speech translation. In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 433–441, Toronto, Canada (in-person and online). Association for Computational Linguistics. find our card
  34. Aiden Williams, Andrea Demarco, and Claudia Borg. 2023b. The applicability of Wav2Vec2 and Whisper for low-resource Maltese ASR. In Proc. 2nd Annual Meeting of the ELRA/ISCA SIG on Under-resourced Languages (SIGUL 2023), pages 39–43. find
  35. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F. Zaidan, and Chris Callison-Burch. 2012. Machine translation of Arabic dialects. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 49–59, Montréal, Canada. Association for Computational Linguistics. find our card
  36. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1628– 1639, Online. Association for Computational Linguistics. find our card
  37. Slavomír Čéplö, Ján Bátora, Adam Benkato, Jiří Mil-ička, Christophe Pereira, and Petr Zemánek. 2016. Mutual intelligibility of spoken Maltese, Libyan Arabic, and Tunisian Arabic functionally tested: A pilot study. Folia Linguistica, 50(2):583–628. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nabhani-2024-um-iwslt-2024-low-resource/