Elamur Research⌕
☰
home / papers / 2024 / UOM-Constrained IWSLT 2024 Shared Task Submission - Maltese Speech Translation

UOM-Constrained IWSLT 2024 Shared Task Submission - Maltese Speech Translation

Summary

This paper describes the University of Malta's entry to the constrained low-resource track of IWSLT 2024. The task was Maltese-to-English speech translation using only a small, fixed amount of training data. The team built a three-stage cascade:

  • ASR: a DeepSpeech 1 recogniser trained on about 12 hours of MASRI and Common Voice audio. An HMM baseline did worse and was dropped.
  • Decoding: a word-level KenLM 3-gram model trained on Korpus Malti v4.0, used during beam-search decoding to pick better transcripts.
  • MT: a Fairseq bidirectional LSTM translation model, chosen over base and large Transformers because it scored best on the development set.

The overall result was very weak: about 0.5 BLEU and a 97.15% word error rate. The authors trace this mainly to the recogniser, whose garbled output looked out-of-domain to a translator trained on clean text. Their error analysis shows typical failure modes:

  • wrong word boundaries
  • spellings that sound right but are wrong
  • mangled named entities

The paper is mainly useful as a documented negative result. It shows how badly errors compound in a cascade when data is scarce, and it suggests pretrained models as the likely remedy.

Key points

  • ASR data and settings: MASRI (6 h 39 min) plus Common Voice (5 h 11 min). DeepSpeech was shrunk to a layer width of 64 instead of the default 2048, with dropout 0.4, batch size 100 and 250 epochs.

  • ASR results: the HMM baseline reached 112.33% WER and DeepSpeech 97.15% (CV 97.0%, Masri 97.43%).

  • MT dev scores (BLEU / chrF):

    ModelBLEUchrF
    LSTM25.7644.57
    Transformer base24.5444.15
    Transformer large25.2043.57

  • MT setup: a SentencePiece tokenizer was trained from scratch. Optuna-style tuning (Akiba et al.) selected learning rate 0.003, dropout 0.04 and 8522 warm-up updates. The final LSTM trained in about 4 minutes with early stopping.

  • Final pipeline BLEU: CV 0.6, Masri 0.2, overall 0.5. KenLM gave only marginal changes (CV BLEU 0.48 → 0.52; Masri chrF 14.50 → 14.73).

  • Code: the authors released their code on GitHub (melanie-galea/uom_constrained).

Interesting ideas

  • I1. When data is tiny, shrinking the network drastically (64 units instead of 2048) is a deliberate guard against overfitting rather than a compromise, and it is worth trying before reaching for bigger models. — For Mova (learn): When training small models on silver data for rare constructions, start from deliberately narrow configurations and only grow them if dev scores justify it.
  • I2. An older, simpler architecture (a single-layer BiLSTM) beat Transformers in the low-data regime, so architectures should be compared under the actual data budget rather than assumed. — For Mova (tree): Benchmark the perceptron parser against gparse on small or converted treebanks, since the cheaper model may win where annotated data is thin.
  • I3. The authors chose word-level over character-level n-grams by inspecting what each model actually changed on a sample of erroneous outputs, which is a cheap qualitative way to choose a correction model. — For Mova (tokens): Use a word n-gram model to rescore competing tokenization or spelling candidates, and choose its granularity by reviewing its edits on a small error sample.
  • I4. The translator, trained on clean text, treated noisy recogniser output as out-of-domain, and the cascade made it impossible to judge the MT component on its own. — For Mova (answer): Evaluate readers on both gold UD trees and predicted trees, so that answer failures can be attributed to parsing errors or to reasoning errors.
  • I5. The error analysis sorted failures into types (boundary merges/splits, phonetically plausible misspellings, named entities), which says far more than a single WER figure. — For Mova (learn): Keep a typed error taxonomy for each pipeline stage, and use it to target self-induced repair rules at the most frequent failure class.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Speech & spoken language · Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.iwslt-1.15

@inproceedings{abela2024uom,
  title = {{UOM-Constrained IWSLT 2024 Shared Task Submission - Maltese Speech Translation}},
  author = {Abela, Kurt and Riyadh, Md Abdur Razzaq and Galea, Melanie and Busuttil, Alana and Kovalev, Roman and Williams, Aiden and Borg, Claudia},
  booktitle = {Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.iwslt-1.15},
  url = {https://doi.org/10.18653/v1/2024.iwslt-1.15}
}

References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. Preprint, arXiv:1907.10902. ↗ our card
  2. Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse Engel, Linxi Fan, Christopher Fougner, Tony Han, Awni Hannun, Billy Jun, Patrick LeGresley, Libby Lin, Sharan Narang, Andrew Ng, Sherjil Ozair, Ryan Prenger, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Yi Wang, Zhiqian Wang, Chong Wang, Bo Xiao, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. 2015. Deep speech 2: End-to-end speech recognition in english and mandarin. Preprint, arXiv:1512.02595. ↗ our card
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. ↗ our card
  4. Liang Ding and Dacheng Tao. 2021. The USYD-JD speech translation system for IWSLT2021. In Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021), pages 182–191, Bangkok, Thailand (online). Association for Computational Linguistics. find
  5. John E. Ortega, Rodolfo Zevallos, and William Chen. 2023a. QUESPA submission for the IWSLT 2023 dialect and low-resource speech translation tasks. In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 261–268, Toronto, Canada (in-person and online). Association for Computational Linguistics. find
  6. John E. Ortega, Rodolfo Zevallos, and William Chen. 2023b. QUESPA Submission for the IWSLT 2023 Dialect and Low-resource Speech Translation Tasks. In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 261–268, Toronto, Canada (in-person and online). Association for Computational Linguistics. find
  7. D. Ellis and N. Morgan. 1999. Size matters: an empirical study of neural network training for large vocabulary continuous speech recognition. In 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258), volume 2, pages 1013–1016 vol.2. find
  8. Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng. 2014. Deep speech: Scaling up end-to-end speech recognition. Preprint, arXiv:1412.5567. ↗ our card
  9. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, et al. 2018. Achieving human parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567. ↗ our card
  10. Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland. Association for Computational Linguistics. find our card
  11. Wan-Hua Her and Udo Kruschwitz. 2024. Investigating Neural Machine Translation for Low-Resource Languages: Using Bavarian as a Case Study. arXiv preprint. ArXiv:2404.08259 [cs]. 160 5 ↗
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. ↗
  13. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67–72, Vancouver, Canada. Association for Computational Linguistics. find our card
  14. Honglak Lee, Yan Largman, Peter Pham, and Andrew Y. Ng. 2009. Unsupervised feature learning for audio classification using convolutional deep belief networks. In Proceedings of the 22nd International Conference on Neural Information Processing Systems, NIPS’09, page 1096–1104, Red Hook, NY, USA. Curran Associates Inc. find
  15. Kurt Micallef, Albert Gatt, Marc Tanti, Lonneke van der Plas, and Claudia Borg. 2022. Pre-training data quality and quantity for a low-resource language: New corpus and BERT models for Maltese. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 90–101, Hybrid. Association for Computational Linguistics. find
  16. Michael Rosner and Jo-Ann Bajada. 2007. Phrase extraction for machine translation. Accepted: 2017-10-17T13:30:37Z Publisher: University of Malta. Faculty of ICT. find
  17. Mike Rosner and Jan Joachimsen. 2012. Il-Lingwa Maltija Fl-Era Diġitali – The Maltese Language in the Digital Age. META-NET White Paper Series. Georg Rehm and Hans Uszkoreit (Series Editors). Springer. find
  18. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning internal representations by error propagation, parallel distributed processing, explorations in the microstructure of cognition, ed. de rumelhart and j. mcclelland. vol. 1. 1986. Biometrika, 71:599–607. find
  19. Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised Pre-Training for Speech Recognition. In Proc. Interspeech 2019, pages 3465–3469. find
  20. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27. find
  21. Shaveta Nagpal, Munish Kumar, · Maruthi, Rohit Ayya-gari, and · Kumar. 2019. A survey of deep learning and its applications: A new paradigm to machine learning. Archives of Computational Methods in Engineering. find
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. find our card
  23. Antoine Nzeyimana. 2024. Low-resource neural machine translation with morphological modeling. arXiv preprint. ArXiv:2404.02392 [cs]. ↗
  24. Aiden Williams, Kurt Abela, Rishu Kumar, Martin Bär, Hannah Billinghurst, Kurt Micallef, Ahnaf Mozib Samin, Andrea DeMarco, Lonneke Van Der Plas, and Claudia Borg. 2023. UM-DFKI Maltese Speech Translation. In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 433–441, Toronto, Canada (in-person and online). Association for Computational Linguistics. find our card
  25. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. find our card
  26. Vineel Pratap, Awni Hannun, Qiantong Xu, Jeff Cai, Jacob Kahn, Gabriel Synnaeve, Vitaliy Liptchinsky, and Ronan Collobert. 2019. Wav2letter++: A fast open-source speech recognition system. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6460–6464. find
  27. Michael Przystupa and Muhammad Abdul-Mageed. 2019. Neural machine translation of low-resource and similar languages with backtranslation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 224–235. find
  28. L.R. Rabiner. 1989. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257–286. find
  29. Alec Radford, Luke Metz, and Soumith Chintala. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. Preprint, arXiv:1511.06434. ↗ our card
  30. Marcely Zanon Boito, John Ortega, Hugo Riguidel, Antoine Laurent, Loïc Barrault, Fethi Bougares, Firas Chaabani, Ha Nguyen, Florentin Barbier, Souhir Gahbiche, and Yannick Estève. 2022. ON-TRAC consortium systems for the IWSLT 2022 dialect and low-resource speech translation tasks. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 308–318, Dublin, Ireland (in-person and online). Association for Computational Linguistics. find
  31. Weitai Zhang, Zhongyi Ye, Haitao Tang, Xiaoxi Li, Xinyuan Zhou, Jing Yang, Jianwei Cui, Pan Deng, Mohan Shi, Yifan Song, Dan Liu, Junhua Liu, and Lirong Dai. 2022. The USTC-NELSLIP offline speech translation systems for IWSLT 2022. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 198–207, Dublin, Ireland (in-person and online). Association for Computational Linguistics. 161 6 find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abela-2024-uom-constrained-iwslt-2024-shared/