Elamur Research⌕
☰
home / papers / 2018 / Tied Multitask Learning for Neural Speech Translation

Tied Multitask Learning for Neural Speech Translation

Summary

Endangered-language documentation often yields only small speech collections, each recording paired with a transcription and a translation into a better-resourced language. Common multitask sequence-to-sequence setups (e.g., Weiss et al., 2017) let the transcription and translation decoders share only the speech encoder. Anastasopoulos and Chiang argue that translation should draw on a more abstract representation. They propose a "triangle" model in which the translation decoder attends both to the encoder's frame states and to the hidden states of the transcription decoder, with the two context vectors concatenated. They also add attention regularizers that push the attention matrices toward transitivity and invertibility, and they decode with a two-phase beam search. On Spanish-English, Mboshi-French and Ainu-English speech, the triangle model outperforms single-task, ASR→MT pivot, standard multitask and cascade baselines in almost all cases. It improves transcription by up to 5% CER and translation by up to 2.8 character-level BLEU. Attention-based word discovery also gains about 3 F-score. Similar gains did not appear in high-resource text translation (English, French, German).

Key points

  • Speech data:
    • CALLHOME Spanish (LDC2014T23) with English translations: about 20 h, more than 240 speakers.
    • Mboshi-French corpus (Godard et al., 2017): about 4.4 h, three speakers.
    • Glossed Audio Corpus of Ainu Folklore: about 2.5 h, one speaker, evaluated with cross-validation.
  • Example results (vs. single-task):
    • Mboshi CER fell from 42.3 to 31.8.
    • Spanish→English BLEU rose from 21.6 to 28.8 (triangle model, 1-best search).
  • Training objective: an equal-weight (λ = 0.5) log-linear mix of the two decoders' likelihoods. The transitivity and invertibility regularizers each have weight 0.2.
  • Implementation: built in DyNet with dropout 0.2, Adam (learning rate 0.0002), beam size 4 per decoder and length normalization 0.8. Code is available on Bitbucket (antonis/dynet-multitask-models).
  • Model size: the model is much smaller than Weiss et al.'s 8-layer-encoder system, which was trained on about 163 h. This suits extremely low-resource settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

79 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 75 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1008

@inproceedings{anastasopoulos2018tied,
  title = {{Tied Multitask Learning for Neural Speech Translation}},
  author = {Anastasopoulos, Antonios and Chiang, David},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1008},
  url = {https://doi.org/10.18653/v1/n18-1008}
}

References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gilles Adda, Sebastian Stüker, Martine Adda-Decker, Odette Ambouroue, Laurent Besacier, David Blachon, Hélène Bonneau-Maynard, Pierre Godard, Fatima Hamlaoui, Dmitry Idiatov, et al. 2016. Breaking the unwritten language barrier: The BULB project. Procedia Computer Science, 81:8–14. find
  2. Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, and Trevor Cohn. 2016. An attentional model for speech translation without transcription. In Proc. NAACL HLT. find
  3. Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. Learning to parse and translate improves neural machine translation. In Proc. ACL. find our card
  4. Antonios Anastasopoulos, David Chiang, and Long Duong. 2016. An unsupervised probability model Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In Proc. AAAI. find
  5. P. Godard, G. Adda, M. Adda-Decker, J. Benjumea, L. Besacier, J. Cooper-Leavitt, G-N. Kouarata, L. Lamel, H. Maynard, M. Mueller, et al. 2017. A very low resource language speech corpus for computational language documentation experiments. arXiv:1710.03501. find
  6. Haifeng Wang, Hua Wu, and Zhanyi Liu. 2006. Word alignment for languages with scarce resources using bilingual corpora of other language pairs. In Proc. COLING/ACL, pages 874–881. find
  7. Hynek Hermansky. 1990. Perceptual linear predictive (PLP) analysis of speech. J. Acoustical Society of America, 87(4):1738–1752. find
  8. Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly transcribe foreign speech. In Proc. INTERSPEECH. find
  9. Aren Jansen, Kenneth Church, and Hynek Hermansky. 2010. Towards spoken term discovery at scale with zero resources. In Proc. INTERSPEECH. find
  10. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144. find
  11. Suyoun Kim, Takaaki Hori, and Shinji Watanabe. 2017. Joint CTC-attention based end-to-end speech recognition using multi-task learning. In Proc. ICASSP. find
  12. Tomer Levinboim and David Chiang. 2015. Multi-task word alignment triangulation for low-resource languages. In Proc. NAACL HLT. find
  13. Tomer Levinboim, Ashish Vaswani, and David Chiang. 2015. Model invertibility regularization: Sequence alignment with or without parallel data. In Proc. NAACL HLT. find
  14. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In Proc. ICLR. find our card
  15. Evgeny Matusov, Stephan Kanthak, and Hermann Ney. 2005. On the integration of speech recognition and statistical machine translation. In Ninth European Conference on Speech Communication and Technology. find
  16. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. DyNet: The dynamic neural network toolkit. arXiv:1701.03980. find our card
  17. Hermann Ney. 1999. Speech translation: Coupling of recognition and translation. In Proc. ICASSP, volume 1. find
  18. Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource related languages for neural machine translation. In Proc. IJCNLP. find our card
  19. Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2013. Improved speech-to-text translation with the Fisher and Callhome Spanish-English speech translation corpus. In Proc. IWSLT. find
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. ACL. find
  21. Shubham Toshniwal, Hao Tang, Liang Lu, and Karen Livescu. 2017. Multitask learning with low-level auxiliary tasks for encoder-decoder based speech recognition. In Proc. Interspeech. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/anastasopoulos-2018-tied-multitask-learning-neural-speech/