Elamur Research⌕
☰
home / papers / 2020 / CoVoST 2 and Massively Multilingual Speech-to-Text Translation

CoVoST 2 and Massively Multilingual Speech-to-Text Translation

Paper · 2020 · Changhan Wang, Anne Wu and Juan Pino

Summary

Speech-to-text translation (ST) research has relied on benchmarks such as MuST-C, Europarl-ST and the original CoVoST. These cover few languages and are mostly European and high-resource. CoVoST 2 tackles this gap. It extends CoVoST, which is built on Mozilla's Common Voice recordings, into a corpus with 21 source languages translated into English and English translated into 15 target languages. Professional translators produced the translations. The authors filtered them with three checks: language-model perplexity (Transformer models trained on OSCAR text), LASER similarity scores and a length-ratio heuristic. Suspicious outputs were reviewed by hand and re-translated where needed. The authors also defined new "CoVoST splits" that reuse recordings Common Voice had discarded, while keeping speakers separate across splits. They trained baselines in fairseq for ASR, MT, cascaded ST and end-to-end ST, in bilingual and multilingual settings. Quality depends strongly on how much training data each language pair has. Multilingual training and data from the reverse direction both look useful for low-resource pairs. The corpus is released under CC0, making it the largest open ST resource at the time.

Key points

  • Corpus size: total speech duration grows from about 700 hours in CoVoST to 2,880 hours in CoVoST 2. Speaker count rises from roughly 11K to 78K, spread over 66 accent groups.
  • CoVoST splits: these raise data utilization from 44.2% (1,273 h) to 78.8% (2,270 h). Training uses the CoVoST train split, and evaluation uses the original Common Voice dev/test splits.
  • Model architecture: ASR and ST share a Transformer with 12 encoder and 6 decoder layers plus a convolutional downsampler. Multilingual decoding is steered by a target-language ID token.
  • Training setup: ST and non-English ASR encoders are pre-trained from English or multilingual ASR. Multilingual settings are All→En, En→All and All→All.
  • Resource gap: monolingual ASR WER is about 18 for French but around 81 for Tamil, which shows how much results depend on data volume. Recipes and data are public on GitHub (fairseq, facebookresearch/covost).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States214 works · 241,492 citations🏢Meta (United Kingdom)London · United Kingdom54 works · 19,348 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 worksSpeech & spoken language · 519 works

Cite

Cite as Changhan Wang, Anne Wu and Juan Pino. 2020. CoVoST 2 and Massively Multilingual Speech-to-Text Translation. arXiv:2007.10310.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2007.10310 · arXiv 2007.10310

@misc{wang2020covost,
  title = {{CoVoST 2 and Massively Multilingual Speech-to-Text Translation}},
  author = {Changhan Wang, Anne Wu and Juan Pino},
  year = {2020},
  doi = {10.48550/arxiv.2007.10310},
  eprint = {2007.10310},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2007.10310}
}

References

22 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 4218–4…
  2. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges.
  3. Mikel Artetxe and Holger Schwenk. 2019. Margin-based parallel corpus mining with multilingual sentence embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3197–3203, Florence, Italy. Association for Computational Linguistics.
  4. Alexandre Berard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text translation. In Proceedings of the 2016 NeurIPS Workshop on End-to-end Learning for Speech and Audio Processing. find
  5. M. A. Di Gangi, M. Negri, and M. Turchi. 2019. Oneto-many multilingual end-to-end speech translation. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 585–592. find
  6. Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (…
  7. H. Inaguma, K. Duh, T. Kawahara, and S. Watanabe. 2019. Multilingual end-to-end speech translation. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 570–577. find
  8. J. Iranzo-Sánchez, J. A. Silvestre-Cerdà, J. Jorge, N. Roselló, A. Giménez, A. Sanchis, J. Civera, and A. Juan. 2020. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing… find
  9. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ…
  10. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR’s WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 314–319, Florence, Italy. Association for Computat…
  11. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoı̂t Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online. Association for Computational…
  12. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
  13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.
  14. Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779.
  15. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics.
  16. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. 2011. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding. IEEE Signal Processing So… find
  17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li…
  18. Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. 2020. End-to-end asr: from supervised to semi-supervised learning with mo Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,… find
  19. Changhan Wang, Anirudh Jain, Danlu Chen, and Jiatao Gu. 2019. Vizseq: A visual analysis toolkit for text generation tasks. find
  20. EMNLP-IJCNLP 2019, page 253. find
  21. Changhan Wang, Juan Pino, Anne Wu, and Jiatao Gu. 2020a. CoVoST: A diverse multilingual speech-to-text translation corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 4197–4203, Marseille, France. European Language Resources Association. find
  22. Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020b. fairseq s2t: Fast speech-to-text modeling with fairseq. arXiv preprint arXiv:2010.05171.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2020-covost-2-massively-multilingual-speech/