Elamur Research⌕
☰
home / papers / 2021 / Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?

Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?

Paper · 2021 · Luisa Bentivogli, Mauro Cettolo, Marco Gaido et al.

Summary

The paper asks whether direct (end-to-end) speech translation has caught up with the traditional cascade pipeline of ASR followed by MT. Earlier comparisons relied on unevenly optimized systems, small training sets and single-reference automatic metrics. To avoid these weaknesses, the authors built state-of-the-art Transformer-based cascade and direct systems for English→German, Spanish and Italian. They evaluated both on carefully checked subsets of the TED-based MuST-C corpus, using several automatic metrics together with professional post-edits and annotations.

The analysis covers:

  • overall quality,
  • how performance relates to properties of the input audio,
  • lexical, morphological and word-order errors,
  • whether humans or an automatic classifier can tell the two outputs apart.

The two paradigms now perform essentially on par. The cascade keeps a slight edge in morphology, word ordering and lexical diversity, while direct models do better at understanding audio and capturing prosody. These differences are too small and inconsistent across languages for humans to reliably distinguish or prefer either system. The authors also release their post-edits, and they caution that the findings may not carry over to distant languages or specialized domains.

Key points

  • On the en-de MuST-C Common test set, the cascade scores 28.9 BLEU and the direct model 29.1, comparable to the IWSLT-20 winning cascade ensemble (28.8). The cascade's ASR component reaches 10.2 WER.
  • Each evaluation set (PE-set) contains 550 manually verified segments, about 10,000 English source words. Two professional translators per language post-edited it, following the IWSLT 2013–2017 methodology.
  • Human assessors could not separate the systems for en-es and en-it, identifying at most 4 of 10 blocks correctly. For en-de they mostly succeeded, because the direct output mirrors English sentence structure. This matches a +19.6% concentration of word-order errors for the direct en-de system.
  • A 5-gram language model + Naive Bayes classifier reaches about 80% accuracy once it sees blocks of 20 sentences, which points to system-specific fluency "fingerprints". Lexical diversity (maTTR, MTLD) is higher for the cascade in all three directions.
  • The post-edits are publicly available at ict.fbk.eu/mustc-post-edits.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TrentoTrento · Italy62 works · 14,868 citations🔬Fondazione Bruno KesslerTrento · Italy25 works · 43,498 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 233 works

Cite

Cite as Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alberto Martinelli, Matteo Negri and Marco Turchi. 2021. Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.224

@inproceedings{bentivogli2021cascade,
  title = {{Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?}},
  author = {Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alberto Martinelli, Matteo Negri and Marco Turchi},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.224},
  url = {https://doi.org/10.18653/v1/2021.acl-long.224}
}

References

73 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Antonios Anastasopoulos and David Chiang. 2018. Tied Multitask Learning for Neural Speech Translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 82–91, New Orleans, US-LA.
  2. Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondřej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang, Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Xing Shi, Sebastian Stüker, Marco Tur… find
  3. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain Adaptation via Pseudo In-Domain Data Selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Edinburgh, Scotland, UK.
  4. Parnia Bahar, Tobias Bieschke, and Hermann Ney. 2019a. A Comparative Study on End-to-end Speech to Text Translation. In Proceedings of the International Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 792–799, Sentosa, Singapore. find
  5. Parnia Bahar, Patrick Wilken, Tamer Alkhouli, Andreas Guta, Pavel Golik, Evgeny Matusov, and Christian Herold. 2020. Start-Before-End and End-to-End: Neural Speech Translation by AppTek and RWTH Aachen University. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), V… find
  6. Parnia Bahar, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019b. On Using SpecAugment for End-to-End Speech Translation. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Hong Kong. find
  7. Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2019. Pre-training on High-resource Speech Recognition Improves Low-resource Speech-to-text Translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAA…
  8. Daniel Beck, Trevor Cohn, and Gholamreza Haffari. 2019. Neural speech translation using lattice transformations and graph networks. In Proceedings of the EMNLP Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), pages 26–31, Hong Kong. find
  9. Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2018a. Neural versus phrase-based MT quality: an in-depth analysis on English–German and English–French. Speech and Language”, 49:52 – 70. find
  10. Computer Luisa Bentivogli, Mauro Cettolo, Marcello Federico, and Christian Federmann. 2018b. Machine Translation Human Evaluation: an investigation of evaluation based on Post-Editing and its relation with Direct Assessment. In Proceedings of the International Conference on Spoken Language Translati… find
  11. Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018. End-to-End Automatic Speech Translation of Audiobooks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6224–6228, Calgary, Canada. find
  12. Alexandre Bérard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation. In Proceedings of the NIPS Workshop on end-to-end learning for speech and audio processing, Barcelona, Spain. find
  13. Nicola Bertoldi and Marcello Federico. 2005. A new decoder for spoken language translation based on confusion networks. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 86–91, San Juan, Puerto Rico. find
  14. Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 Workshop on Statistical Machine Translation. In Proce…
  15. Roldano Cattoni, Mattia A. Di Gangi, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2020. MuST-C: A Multilingual Corpus for end-to-end Speech Translation. Computer Speech & Language Journal. Doi: https://doi.org/10.1016/j.csl.2020.101155. — [doi:10.1016/j.csl.2020.101155](https://doi.org/10.1016/j.csl.2020.101155) ↗
  16. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2013. Report on the 10th IWSLT Evaluation Campaign. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Heidelberg, Germany. find
  17. Qiao Cheng, Meiyuan Fang, Yaqian Han, Jin Huang, and Yitao Duan. 2019. Breaking the data barrier: Towards robust speech translation via adversarial stability training. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Hong Kong. find
  18. Michael A. Covington and Joe D. McFall. 2010. Cutting the Gordian Knot: The Moving-Average Type–Token Ratio (MATTR). Journal of Quantitative Linguistics, 17(2):94–100. find
  19. Michael Denkowski and Alon Lavie. 2010. Choosing the right evaluation for machine translation: An examination of annotator and automatic metric performance on human judgment tasks. In Proceedings of the Conference of the Association of Machine Translation in the Americas (AMTA), Denver, US-CO. find
  20. Mattia A. Di Gangi, Robert Enyedi, Alessandra Brusadin, and Marcello Federico. 2019a. Robust Neural Machine Translation for Clean and Noisy Speech Transcripts. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Hong Kong. find
  21. Mattia A. Di Gangi, Matteo Negri, and Marco Turchi. 2019b. Adapting Transformer to End-to-end Spoken Language Translation. In Proceedings of the Conference of the International Speech Communication Association (INTERSPEECH), Graz, Austria. find
  22. Mattia Antonino Di Gangi, Matteo Negri, and Marco Turchi. 2019c. One-to-many multilingual end-to-end speech translation. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 14-18 December 2019, Sentosa, Singapore. find
  23. Marco Gaido, Mattia A. Di Gangi, Matteo Negri, and Marco Turchi. 2020. End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020. In Proceedings of the International Conference on Spoken Language Translation (IWSLT), Virtual Event. find
  24. Yvette Graham, Timothy Baldwin, Meghan Dowling, Maria Eskevich, Teresa Lynn, and Lamia Tounsi. 2016. Is all that Glitters in Machine Translation Quality Estimation really Gold? In Proceedings of the International Conference on Computational Linguistics (COLING), pages 3124–3134, Osaka, Japan. find
  25. Steven Gutstein, Olac Fuentes, and Eric Freudenthal. 2008. Knowledge Transfer in Deep Convolutional Neural Nets. International Journal on Artificial Intelligence Tools, 17(03):555–567. find
  26. François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Estève. 2018. TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation. In Proceedings of the Speech and Computer - 20th International Conference (SPECOM), pages 198– 208, Leipzig,… find
  27. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. In Proceedings of NIPS Deep Learning and Representation Learning Workshop, Montréal, Canada.
  28. Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, and Shinji Watanabe. 2020. ESPnet-ST: All-in-One Speech Translation Toolkit. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL): System Demonstrations, Virtual Event. find
  29. Sathish R. Indurthi, Houjeung Han, Nikhil K. Lakumarapu, Beomseok Lee, Insoo Chung, Sangha Kim, and Chanwoo Kim. 2020. End-end Speech-to-Text Translation with Modality Agnostic Meta-Learning. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pag… find
  30. Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerdà, Javier Jorge, Nahuel Roselló, Giménez. Adrià, Albert Sanchis, Jorge Civera, and Alfons Juan. 2020. Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debates. In Proceedings of the IEEE International Conference on Acous… find
  31. Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron J. Weiss, Yuan Cao, Chung-Cheng Chiu, Naveen Ari, Stella Laurenzo, and Yonghui Wu. 2019. Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation. In Proceedings of the IEEE International Conference on Acoustics, Speech and Sig… find
  32. Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. 2015. Audio Augmentation for Speech Recognition. In Proceedings of the Conference of the International Speech Communication Association (INTERSPEECH), pages 3586– 3589, Dresden, Germany. find
  33. Philipp Koehn. 2004. Statistical Significance Tests for Machine Translation Evaluation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Barcelona, Spain.
  34. Alon Lavie, Donna Gates, Marsal Gavalda, Laura Mayfield Tomokiyo, Alex Waibel, and Lori Levin. 1996. Multi-lingual translation of spontaneously spoken language in a limited domain. In Proceedings of the International Conference on Computational Linguistics (COLING), Copenhagen, Denmark. find
  35. Yuchen Liu, Hao Xiong, Jiajun Zhang, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019. End-to-End Speech Translation with Knowledge Distillation. In Proceedings of the Conference of the International Speech Communication Association (INTERSPEECH), pages 1128–1132, Graz, Austria. find
  36. Yuchen Liu, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2020. Bridging the Modality Gap for Speech-to-Text Translation. arxiv.org/pdf/2010.14920.pdf. — [arXiv:2010.14920](https://arxiv.org/abs/2010.14920) find
  37. Evgeny Matusov, Hermann Ney, and Ralph Schluter. 2005. Phrase-based translation of speech recognizer word lattices using loglinear model combination. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 110–115, San Juan, Puerto Rico. find
  38. Philip M. McCarthy and Scott Jarvis. 2010. MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2):381–392. find
  39. Graham Neubig, Matthias Sperber, Xinyi Wang, Matthieu Felix, Austin Matthews, Sarguna Padmanabhan, Ye Qi, Devendra Singh Sachan, Philip Arthur, Pierre Godard, John Hewitt, Rachid Riad, and Liming Wang. 2018. XNMT: The extensible neural machine translation toolkit. In Conference of the Association fo… find
  40. Thai-Son Nguyen, Sebastian Stueker, Jan Niehues, and Alex Waibel. 2020. Improving Sequence-to-sequence Speech Recognition Training with On-the-fly Data Augmentation. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Barcelona, Spain. find
  41. Jan Niehues, Roldano Cattoni, Sebastian Stucker, Matteo Negri, Marco Turchi, et al. 2019a. find
  42. The IWSLT 2019 Evaluation Campaign. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Hong Kong. find
  43. Jan Niehues, Roldano Cattoni, Sebastian Stüker, Mauro Cettolo, Marco Turchi, and Marcello Federico. 2018. find
  44. The IWSLT 2018 Evaluation Campaign. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Bruges, Belgium. find
  45. Jan Niehues, Roldano Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Thanh-Le ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico. 2019b. find
  46. The IWSLT 2019 Evaluation Campaign. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Hong Kong. find
  47. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr…
  48. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an ASR Corpus Based on Public Domain Audio Books. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210, Brisbane, Australia. find
  49. Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. In Proceedings of the Conference of the International Speech Communication Association (IN-TERSPEECH), pages 26…
  50. Stephan Peitz, Simon Wiesler, Markus Nußbaum-Thom, and Hermann Ney. 2012. Spoken language translation using automatically transcribed text in training. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Hong Kong. find
  51. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the Conference of the Association for Machine Translation of the Americas (AMTA), pages 223–231, Cambridge, US-MA.
  52. Fuchun Peng and Dale Schuurmans. 2003. Combining Naive Bayes and n-Gram Language Models for Text Classification. In Proceedings of the European Conference on Information Retrieval (ECIR), pages 335–350, Pisa, Italy. find
  53. Matthew Snover, Nitin Madnani, Bonnie J Dorr, and Richard Schwartz. 2009. Fluency, Adequacy, or HTER?: exploring different human judgments with a tunable MT metric. In Proceedings of the Workshop on Statistical Machine Translation (WMT), pages 259–268, Athens, Greece.
  54. Ngoc-Quan Pham, Thai-Son Nguyen, Thanh-Le Ha, Juan Hussain, Felix Schneider, Jan Niehues, Sebastian Stüker, and Alexander Waibel. 2019. find
  55. The IWSLT 2019 KIT Speech Translation System. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Hong Kong. find
  56. Matthias Sperber, Graham Neubig, Ngoc-Quan Pham, and Alex Waibel. 2019. Self-attentional models for lattice inputs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 1185–1197, Florence, Italy. find
  57. Juan Pino, Liezl Puzon, Jiatao Gu, Xutai Ma, Arya D. McCarthy, and Deepak Gopinath. 2019. Harnessing Indirect Training Data for End-to-End Automatic Speech Translation: Tricks of the Trade. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Hong Kong. find
  58. Matthias Sperber, Jan Niehues, Graham Neubig, Sebastian Stüker, and Alex Waibel. 2018. Self-Attentional Acoustic Models. In Proceedings of the Conference of the International Speech Communication Association (INTERSPEECH), pages 3723–3727, Hyderabad, India. find
  59. Matt Post. 2018. A Call for Clarity in Reporting BLEU Scores. In Proceedings of the Conference on Machine Translation (WMT), pages 186–191, Brussels, Belgium.
  60. Matthias Sperber, Jan Niehues, and Alex Waibel. 2017. Toward Robust Neural Machine Translation for Noisy Input Sequences. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Tokyo, Japan. find
  61. Tomasz Potapczyk and Pawel Przybysz. 2020. SR-POL’s System for the IWSLT 2020 End-to-End Speech Translation Task. In Proceedings of the International Conference on Spoken Language Translation (IWSLT), Virtual Event. find
  62. Daniel Povey, Hossein Hadian, Pegah Ghahremani, Ke Li, and Sanjeev Khudanpur. 2018. A Time-Restricted Self-Attention Layer for ASR. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5874–5878, Calgary, Canada. find
  63. Nicholas Ruiz, Qin Gao, William Lewis, and Marcello Federico. 2015. Adapting machine translation models toward misrecognized speech with text-to-speech pronunciation rules and acoustic confusability. In Proceedings of the Conference of the International Speech Communication Association (INTER-SPEECH… find
  64. Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loı̈c Barrault, Lucia Specia, and Florian Metze. 2018. How2: A Large-scale Dataset For Multimodal Language Understanding. In Proceedings of Visually Grounded Interaction and Language (ViGIL), Montréal, Canada. Neural Information Proce… find
  65. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural Machine Translation of Rare Words with Subword Units. arXiv preprint arXiv:1508.07909.
  66. Matthias Sperber and Matthias Paulik. 2020. Speech translation and the end-to-end promise: Taking stock of where we are. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 7409–7421, Virtual Event.
  67. Fred W.M. Stentiford and Martin G. Steer. 1988. Machine translation of speech. British Telecom Technology Journal, 6(2):116–122. find
  68. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the Inception Architecture for Computer Vision. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, Las Vegas, Nevada, United States. find
  69. Jörg Tiedemann. 2016. Opus – parallel corpora for everyone. Baltic Journal of Modern Computing, page 384. Special Issue: Proceedings of the 19th Annual Conference of the European Association of Machine Translation (EAMT). find
  70. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In Proceedings of Advances in Neural Information Processing Systems 30 (NIPS), pages 5998–6008, Long Beach, US-CA.
  71. Alex Waibel, Ajay N Jain, Arthur E McNair, Hiroaki Saito, Alexander G Hauptmann, and Joe Tebelskis. 1991. Janus: a speech-to-speech translation system using connectionist and symbolic processing strategies. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processi… find
  72. Andy Way. 2018. Quality Expectations of Machine Translation. In S. Castilho, J. Moorkens, F. Gaspari, and S. Doherty, editors, Translation quality assessment: From Principles to Practice, pages 159–178. Springer. find
  73. Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-Sequence Models Can Directly Translate Foreign Speech. In Proceedings of the Conference of the International Speech Communication Association (IN-TERSPEECH), pages 2625–2629, Stockholm, Sweden. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bentivogli-2021-cascade-versus-direct-speech-translation/