Elamur Research⌕
☰
home / papers / 2019 / Pre-training on high-resource speech recognition improves low-resource speech-to-text translation

Pre-training on high-resource speech recognition improves low-resource speech-to-text translation

Paper · 2019 · Sameer Bansal, Herman Kamper, Karen Livescu et al.

Summary

End-to-end speech-to-text translation (ST) works well when large amounts of translated audio exist, but it performs poorly when only a few hours are available. This is typical for endangered or unwritten languages. Bansal et al. propose a simple form of transfer learning. They first train an attention-based encoder-decoder (CNN + bi-LSTM encoder over MFCCs, LSTM decoder over BPE subwords) for automatic speech recognition (ASR) in a well-resourced language. They then reuse some or all of its weights to initialise the ST model and fine-tune on the small parallel set. For Spanish-English, they used Fisher subsets with Switchboard English ASR. For Mboshi-French, they used English and GlobalPhone French ASR. On 20 hours of Spanish-English, the method nearly doubles BLEU. Ablations show the gain comes mostly from the transferred acoustic encoder. As a result, ASR data in an unrelated third language still helps. The authors conclude that acoustic-phonetic modelling, more than translation itself, is a central bottleneck for low-resource ST.

Key points

  • Pre-training on about 300 hours of English Switchboard ASR (en-300h) lifts Spanish-English ST trained on 20 hours of Fisher data from 10.8 to 20.2 BLEU.
  • Transferring only the encoder delivers most of the gain, even when the ASR language matches neither ST language. On the dev set, a French encoder (fr-20h, GlobalPhone) raises BLEU from 10.8 to 12.5, and an English encoder trained on 20 hours reaches 13.2.
  • On Mboshi-French (Godard et al., 2018 corpus, ~4 hours), combining the en-300h encoder with the fr-20h attention and decoder raises BLEU from 3.5 to 7.1, with precision/recall around 25%. The authors believe these are the first word-level BLEU results reported for this corpus.
  • Pre-trained models converge within a couple of hours, while the baseline needs over a day. Training used Chainer on a single Titan X GPU, and the code is publicly released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations🎓Toyota Technological Institute at ChicagoChicago · United States29 works · 10,273 citations🎓Stellenbosch UniversityStellenbosch · South Africa15 works · 24,115 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Signal Processing · 213 works

🧭 Topics 2 Speech & spoken language · 233 works Low-resource languages · 166 works

Cite

Cite as Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez and Sharon Goldwater. 2019. Pre-training on high-resource speech recognition improves low-resource speech-to-text translation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1006

@inproceedings{bansal2019pre,
  title = {{Pre-training on high-resource speech recognition improves low-resource speech-to-text translation}},
  author = {Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez and Sharon Goldwater},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1006},
  url = {https://doi.org/10.18653/v1/n19-1006}
}

References

47 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pierre Godard, Gilles Adda, Martine Adda-Decker, Juan Benjumea, Laurent Besacier, Jamison Cooper-Leavitt, Guy-Noël Kouarata, Lori Lamel, Hélène Maynard, Markus Müller, Annie Rialland, Sebastian Stüker, François Yvon, and Marcely Zanon Boito. 2018. A very low resource language speech corpus for… find
  2. Oliver Adams, Graham Neubig, Trevor Cohn, and Steven Bird. 2016a. Learning a translation model from word lattices. In Proc. Interspeech. find
  3. John Godfrey and Edward Holliman. 1993. Switchboard-1 Release 2 (LDC97S62). https://catalog.ldc.upenn.edu/ldc97s62. ↗
  4. Oliver Adams, Graham Neubig, Trevor Cohn, Steven Bird, Quoc Truong Do, and Satoshi Nakamura. 2016b. Learning a lexicon and translation model from phoneme lattices. In Proc. EMNLP. find
  5. David Graff, Shudong Huang, Ingrid Cartagena, Kevin Walker, and Christopher Cieri. 2010. Fisher Spanish Speech (LDC2010S01). https://catalog.ldc.upenn.edu/ldc2010s01. ↗
  6. Tanel Alumäe, Stavros Tsakalidis, and Richard M Schwartz. 2016. Improved multilingual training of stacked neural network acoustic models for low resource languages. In Proc. Interspeech. find
  7. Caglar Gülçehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loıc Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. CoRR, abs/1503.03535. our card
  8. Antonios Anastasopoulos and David Chiang. 2017. A case study on using speech-to-translation alignments for language documentation. In Proc. ACL. find
  9. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proc. ICCV. find
  10. Antonios Anastasopoulos and David Chiang. 2018. Tied multitask learning for neural speech translation. In Proc. NAACL HLT. our card
  11. Enno Hermann and Sharon Goldwater. 2018. Multilingual bottleneck features for subword modeling in zero-resource languages. In Proc. Interspeech. find
  12. Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2018. Low-resource speech-to-text translation. In Proc. Interspeech. find
  13. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput. find
  14. Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018. End-to-end automatic speech translation of audiobooks. In Proc. ICASSP. find
  15. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. ICML. find
  16. Laurent Besacier, Bowen Zhou, and Yuqing Gao. 2006. Towards speech translation of non written languages. In Proc. SLT. find
  17. Aren Jansen, Kenneth Church, and Hynek Hermansky. 2010. Towards spoken term discovery at scale with zero resources. In Proc. Interspeech. find
  18. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. 2011. The Kaldi Speech Recognition Toolkit. In Proc. ASRU. find
  19. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Trans. ACL, 5… find
  20. Prajit Ramachandran, Peter J Liu, and Quoc V Le. 2017. Unsupervised pretraining for sequence to sequence learning. In Proc. EMNLP. our card
  21. Herman Kamper, Micha Elsner, Aren Jansen, and Sharon Goldwater. 2015. Unsupervised neural network based feature extraction using weak top-down constraints. In Proc. ICASSP. find
  22. Daniel Renshaw, Herman Kamper, Aren Jansen, and Sharon Goldwater. 2015. A comparison of neural network methods for unsupervised representation learning on the zero resource speech challenge. In Proc. Interspeech. find
  23. Herman Kamper, Aren Jansen, and Sharon Goldwater. 2017. A segmental framework for fully-unsupervised large-vocabulary speech recognition. Comput. Speech Lang., 46:154–174. find
  24. Tanja Schultz. 2002. Globalphone: a multilingual speech and text database developed at karlsruhe university. In Seventh International Conference on Spoken Language Processing. find
  25. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. ICLR. find
  26. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proc. ACL. find
  27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. ACL. our card
  28. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. find
  29. Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proc. WMT. our card
  30. Samuel Thomas, Sriram Ganapathy, and Hynek Hermansky. 2012. Multilingual mlp features for low-resource LVCSR systems. In Proc. ICASSP. find
  31. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. EMNLP. our card
  32. Sebastian Thrun. 1995. Is learning the n-th thing any easier than learning the first? In Proc. NIPS. find
  33. Lara J Martin, Andrew Wilkinson, Sai Sumanth Miryala, Vivian Robison, and Alan W Black. 2015. Utterance classification in speech-to-speech translation for zero-resource languages in the hospital administration domain. In Proc. ASRU. find
  34. Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. 2015. Chainer: A next-generation open source framework for deep learning. In Proc. LearningSys. find
  35. Robert Munro. 2010. Crowdsourced translation for emergency response in Haiti: The global collaboration of local knowledge. In AMTA Workshop Collaborative Crowdsourcing Transl. find
  36. Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu, Yonghui Wu, Tara N Sainath, and Karen Livescu. 2018a. A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition. In Proc. SLT. find
  37. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In Proc. ICML. find
  38. Shubham Toshniwal, Tara N. Sainath, Ron J. Weiss, Bo Li, Pedro Moreno, Eugene Weinstein, and Kanishka Rao. 2018b. Multilingual Speech Recognition with A Single End-To-End Model. In Proc. ICASSP. find
  39. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL. our card
  40. Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2013. Improved speech-to-text translation with the Fisher and Callhome Spanish-English speech translation corpus. In Proc. IWSLT. find
  41. Ngoc Thang Vu, Wojtek Breiter, Florian Metze, and Tanja Schultz. 2012. An investigation on initialization schemes for multilayer perceptron training using multilingual data and their effect on ASR performance. In Proc. Interspeech. find
  42. Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2014. Fisher and CALLHOME Spanish– English Speech Translation. https://catalog.ldc.upenn.edu/ldc2014t23. ↗
  43. Ron J Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly transcribe foreign speech. In Proc. Interspeech. find
  44. Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Comput. find
  45. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  46. Yougen Yuan, Cheung-Chi Leung, Lei Xie, Bin Ma, and Haizhou Li. 2016. Learning neural network representations using cross-lingual bottleneck features with word-pair information. In Proc. Interspeech. find
  47. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proc. EMNLP. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bansal-2019-pre-training-high-resource-speech/