Elamur Research⌕
☰
home / papers / 2020 / SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation

SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation

Paper · 2020 · Xutai Ma, Juan Pino and Philipp Koehn

Summary

Methods for simultaneous text translation (SimulMT) decide token by token when to read input and when to emit output. Speech input has no such natural units, so it is unclear when an end-to-end simultaneous speech translation (SimulST) system should make these decisions. The authors add a pre-decision module that groups encoder states into chunks before a policy chooses READ or WRITE. They test two versions of this module. The fixed version triggers after a set time step. The flexible version uses oracle word or phoneme boundaries. Each is paired with a fixed policy (wait-k) or a flexible one (monotonic multihead attention, MMA), built on an S-Transformer trained on English-German MuST-C.

The authors also revise Average Lagging so that it is measured in time rather than steps and can account for computation time. In their experiments, chunk granularity strongly shapes the latency-quality trade-off. MMA handles fine granularity better than wait-k, and MMA with word-level boundaries performs best. The work offers a practical bridge between SimulMT and SimulST and a more realistic latency metric.

Key points

  • Data and setup:
    • Training used 408 hours of speech and 234k sentences from MuST-C En-De, with results reported on the dev set.
    • Features were 80-dimensional log-mel filterbanks extracted with Kaldi.
    • The vocabulary was a SentencePiece unigram model of 10,000 units.
    • Gentle supplied the word and phoneme alignments.
    • Quality was scored with SacreBLEU.
  • Effect of fixed step size:
    • A step of 120ms gave very poor quality for both wait-k and MMA.
    • A step of 280ms gave a good trade-off and closely matches the average word duration of 271ms.
  • Phoneme boundaries average about 77ms, which proved too fine-grained and hurt translation quality. MMA was more robust to this than wait-k.
  • With the computation-aware metric, the gap between computation-aware and non-computation-aware latency narrows as step size grows, because the bidirectional encoder is recomputed less often.
  • The code is released in fairseq. Future work includes an incremental encoder and a learnable pre-decision module.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🏢Meta (Israel)Tel Aviv · Israel38 works · 32,024 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 480 works

Cite

Cite as Xutai Ma, Juan Pino and Philipp Koehn. 2020. SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.aacl-main.58

@inproceedings{ma2020simulmt,
  title = {{SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation}},
  author = {Xutai Ma, Juan Pino and Philipp Koehn},
  booktitle = {Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.aacl-main.58},
  url = {https://doi.org/10.18653/v1/2020.aacl-main.58}
}

References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ashkan Alinejad, Maryam Siahbani, and Anoop Sarkar. 2018. Prediction improves simultaneous neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3022–3027. find
  2. Ebrahim Ansari, amittai axelrod, Nguyen Bach, Ondřej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang, Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Xing Shi, Sebastian Stüker, Marco Tur… find
  3. FINDINGS OF THE IWSLT 2020 EVALU-ATION CAMPAIGN. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 1–34, Online. Association for Computational Linguistics. find
  4. Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li, and Colin Raffel. 2019. Monotonic infinite lookback attention for simultaneous machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pa…
  5. Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, and George Foster. 2020. Re-translation versus streaming for simultaneous translation. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 220–227, Online. Association for Computational Linguistics. find
  6. Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019a. MuST-C: a Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1…
  7. Mattia Antonino Di Gangi, Matteo Negri, Roldano Cattoni, Roberto Dessi, and Marco Turchi. 2019b. Enhancing transformer for end-to-end speech-to-text translation. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 21– 31, Dublin, Ireland. European Association for Machin… find
  8. Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor OK Li. 2017. Learning to translate in real-time with neural machine translation. In 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, pages 1053–1062. Association for Computational Linguistics (ACL…
  9. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71.
  10. Dieterich Lawson, Chung-Cheng Chiu, George Tucker, Colin Raffel, Kevin Swersky, and Navdeep Jaitly. 2018. Learning hard alignments with variational inference. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5799–5803. IEEE. find
  11. Yuping Luo, Chung-Cheng Chiu, Navdeep Jaitly, and Ilya Sutskever. 2017. Learning online alignments with continuous rewards policy gradient. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2801–2805. IEEE. find
  12. Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2019. STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework. In Proceedings of the…
  13. Xutai Ma, Juan Pino, James Cross, Liezl Puzon, and Jiatao Gu. 2020. Monotonic multihead attention. In International Conference on Learning Representations. find
  14. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.
  15. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics.
  16. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Alvin Grissom II, He He, Jordan Boyd-Graber, John Morgan, and Hal Daumé III. 2014. Don’t until the Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. 2011. The kaldi speech recognition toolkit… find
  17. Colin Raffel, Minh-Thang Luong, Peter J Liu, Ron J Weiss, and Douglas Eck. 2017. Online and linear-time attention by enforcing monotonic alignments. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2837–2846. JMLR. org. find
  18. Yi Ren, Jinglin Liu, Xu Tan, Chen Zhang, Tao QIN, Zhou Zhao, and Tie-Yan Liu. 2020. SimulSpeech: End-to-end simultaneous speech to text translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3787–3796, Online. Association for Computational Ling… find
  19. Baigong Zheng, Renjie Zheng, Mingbo Ma, and Liang Huang. 2019a. Simpler and faster learning of adaptive policies for simultaneous translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proc…
  20. Baigong Zheng, Renjie Zheng, Mingbo Ma, and Liang Huang. 2019b. Simultaneous translation with flexible policy via restricted imitation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5816– 5822, Florence, Italy. Association for Computationa… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2020-simulmt-simulst-adapting-simultaneous-text/