Elamur Research⌕
☰
home / papers / 2021 / Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task

Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task

Paper · 2021 · Yun Tang, Juan Pino, Xian Li et al.

Summary

End-to-end speech translation (ST) depends on paired speech–translation data, which is expensive and scarce. Systems are therefore often built from pretrained ASR and MT modules and trained jointly with related tasks. This Facebook AI paper examines how an auxiliary text-to-text MT task actually affects the primary ST task in such a multitask setup. Starting from models initialized with pretrained ASR and MT components, the authors run two analyses. One measures how far the fine-tuned modules move away from their pretrained versions. The other compares internal representations produced from speech versus text input. They find little negative transfer and clear benefits from sharing more parameters. A noticeable modality gap remains in the top decoder layers, which are also the most important for quality. Building on this, they propose three changes: share the whole decoder and upper encoder layers, initialized from MT; add a cross-attentive regularization (CAR) that aligns speech and text encoder outputs; and distill knowledge online from the co-trained MT branch. On MuST-C EN-DE, EN-FR and EN-ES, ST quality improves by over 2 BLEU against a strong baseline and reaches state-of-the-art results.

Key points

  • Setup: A speech encoder and a text encoder feed one shared decoder, and the uppermost encoder layers are likewise common to both paths. The shared parts are initialized from an MT model pretrained on WMT parallel data. Only the ST path is used at inference.
  • CAR: Speech and text encoder outputs have different lengths, so both are reconstructed to text length. The speech outputs are rebuilt through a cosine-similarity attention over the text outputs, and the text outputs through self-attention. The L2 gap between the two is then penalized, with a stop-gradient applied on the text side.
  • Training objective: The loss combines ST NLL, online KD toward the MT branch's output distribution (weighted by α), CAR (weighted by λ), and MT NLL.
  • Final ST BLEU on MuST-C tst-COMMON: 26.8 for EN-DE, 31.0 for EN-ES and 37.4 for EN-FR.
  • Task-specific components and the MT task: Adding an adapter left BLEU unchanged at 24.7, and dedicated decoder attention lowered it to 24.2, so extra task-specific parts seem unnecessary. The co-trained MT models, which take phoneme input, also gained 0.7–1.6 BLEU from the proposed techniques.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 233 works

Cite

Cite as Yun Tang, Juan Pino, Xian Li, Changhan Wang and Dmitriy Genzel. 2021. Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.328

@inproceedings{tang2021improving,
  title = {{Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task}},
  author = {Yun Tang, Juan Pino, Xian Li, Changhan Wang and Dmitriy Genzel},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.328},
  url = {https://doi.org/10.18653/v1/2021.acl-long.328}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Antonios Anastasopoulos and David Chiang. 2018. Tied multitask learning for neural speech translation. In NAACL-HLT.
  2. Parnia Bahar, Tobias Bieschke, and Hermann Ney. 2019. A comparative study on end-to-end speech to text translation. In ASRU. find
  3. Ankur Bapna, N. Arivazhagan, and Orhan Firat. 2019. Simple, scalable adaptation for neural machine translation. In EMNLP/IJCNLP.
  4. G. Blackwood, Miguel Ballesteros, and T. Ward. 2018. Multilingual neural machine translation with task-specific attention. In COLING. find
  5. Niladri S. Chatterji, Behnam Neyshabur, and H. Sedghi. 2020. The intriguing role of module criticality in the generalization of deep networks. In ICLR. find
  6. Z. Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2018. Gradnorm: Gradient normalization for adaptive loss balancing in deep multi-task networks. In ICML. find
  7. Marco Gaido, Mattia Antonino Di Gangi, Matteo Negri, and Marco Turchi. 2020. End-to-end speech-translation with knowledge distillation: Fbk@iwslt2020. find
  8. Mattia Antonino Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019a. MuST-C: a multilingual speech translation corpus. In NAACL-HLT.
  9. Mattia Antonino Di Gangi, Matteo Negri, and Marco Turchi. 2019b. One-to-many multilingual end-to-end speech translation. In ASRU. find
  10. Geoffrey E. Hinton, Oriol Vinyals, and J. Dean. 2015. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531.
  11. Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2020. Multilingual speech translation with efficient finetuning of pre-trained models. arXiv: Computation and Language. find
  12. Shikun Liu, Edward Johns, and A. Davison. 2019a. End-to-end multi-task learning with attention. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1871–1880. find
  13. Yuchen Liu, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019b. End-to-end speech translation with knowledge distillation. In Interspeech. find
  14. K. Maninis, Ilija Radosavovic, and I. Kokkinos. 2019. Attentive single-tasking of multiple tasks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1851–1860. find
  15. M. McCloskey and N. J. Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109–165. find
  16. Ari S. Morcos, M. Raghu, and S. Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. In NeurIPS. find
  17. Jan Niehues, R. Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Elizabeth Salesky, Ramon Sanabria, Loı̈c Barrault, Lucia Specia, and Marcello Federico. 2019. find
  18. The IWSLT 2019 evaluation campaign. find
  19. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, S. Gross, Nathan Ng, David Grangier, and M. Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL.
  20. D. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. Cubuk, and Q. Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. Interspeech.
  21. Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An adapter-based framework for multi-task cross-lingual transfer. In EMNLP.
  22. J. Pino, Q. Xu, X. Ma, M. Dousti, and Y. Tang. 2020. Self-training for end-to-end speech translation. In Interspeech. find
  23. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics.
  24. M. Raghu, J. Gilmer, J. Yosinski, and Jascha Sohl-Dickstein. 2017. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In NIPS. find
  25. A. Renduchintala, S. Ding, M. Wiesner, and S. Watanabe. 2018. Multi-modal data augmentation for end-to-end asr. In INTERSPEECH. find
  26. Elizabeth Salesky and Alan W Black. 2020. Phone features improve speech translation. In ACL. find
  27. T. Standley, A. Zamir, Dawn Chen, L. Guibas, Jitendra Malik, and S. Savarese. 2020. Which tasks should be learned together in multi-task learning? In ICML. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2021-improving-speech-translation-understanding-learning/