Elamur Research
☰
home / papers / 2021 / VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation

VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.acl-long.80.pdf

Summary

Recent gains from speech pre-training and self-training have relied mostly on English audio. Open multilingual speech data becomes scarce beyond a few major languages, and open corpora of real simultaneous interpretation barely exist. The authors build VoxPopuli from European Parliament event recordings from 2009–2020. Energy-based VAD splits the full-event audio into an unlabeled set of about 400K hours across 23 EU languages. For plenary sessions, official transcripts are matched to the audio in three steps: speaker diarization (pyannote.audio) corrects the timestamps, ASR force-alignment cuts utterances, and a CER filter removes bad segments. This yields about 1.8K transcribed hours. Interpreter audio is decoded with ASR and paired with source sentences by choosing the span whose LASER embedding best matches the source text. The result is 17.3K hours of speech-to-speech data. Experiments show that the unlabeled audio improves self-training for ASR and ST, including on out-of-domain Common Voice data. The weakly labeled alignments prove more useful than a much larger pseudo-labeled set. The corpus matters for less-resourced European languages and for interpretation research. Ukrainian is not covered.

Key points

  • Unlabeled data: about 400K hours in 23 EU languages, with 8K–24K hours per language. Irish is excluded due to too little data. Clips are 15–30 s, and roughly 16% of the audio was dropped as silence.
  • Transcribed data: about 1.8K hours. The statistics table lists 16 source languages, while the abstract says 15. English is the largest share at 543 h. Utterances are capped at 20 s and filtered at a 20% CER threshold. Train/dev/test splits use disjoint speakers in an 18:1:1 duration ratio.
  • Interpretation data: 17.3K hours of aligned oral interpretation between 15 source and 15 target languages. Alignments with a LASER cosine similarity below 0.75 are discarded.
  • Self-training: 0.8K/3K hours of VoxPopuli audio (2013–2020 only) were used to self-train Transformer Large models on top of EuroParl-ST + CoVoST 2 labels. This improved both in-domain (EP) and out-of-domain (CV) results. For ST, it narrowed the gap between end-to-end and cascaded systems.
  • Weakly supervised ST: joint training with about 0.4K hours of weakly labeled alignment data beat self-training with 3K hours of pseudo-labels. Released at github.com/facebookresearch/voxpopuli.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

189 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 179 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.80

@inproceedings{wang2021voxpopuli,
  title = {{VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation}},
  author = {Wang, Changhan and Riviere, Morgane and Lee, Ann and Wu, Anne and Talnikar, Chaitanya and Haziza, Daniel and Williamson, Mary and Pino, Juan and Dupoux, Emmanuel},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.80},
  url = {https://doi.org/10.18653/v1/2021.acl-long.80}
}

References

51 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 4218–4… find our card
  2. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. find our card
  3. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33. find our card
  4. Claudio Bendazzoli, Annalisa Sandrelli, et al. 2005. An approach to corpus-based interpreting studies: developing epic (european parliament interpreting corpus). Proceedings of Challenges of Multidimensional Translation. find
  5. Alan W Black. 2019. Cmu wilderness multilingual speech dataset. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5971–5975. IEEE. find
  6. Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. 2020. pyannote.audio: neural building blocks for speaker diarization. In ICASSP 2020, IEEE International Conference on Acoustics, Spe… find
  7. Roldano Cattoni, Mattia Antonino Di Gangi, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2020. Must-c: A multilingual corpus for end-to-end speech translation. Computer Speech & Language, 66:101155. find
  8. Ronan Collobert, Christian Puhrsch, and Gabriel Synnaeve. 2016. Wav2letter: an end-to-end convnet-based speech recognition system. arXiv preprint arXiv:1609.03193. — [arXiv:1609.03193](https://arxiv.org/abs/1609.03193) find
  9. Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2020. Unsupervised cross-lingual representation learning for speech recognition. find
  10. Mattia A Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. Must-c: a multilingual speech translation corpus. In 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2012–2017. Association f… find our card
  11. Ewan Dunbar, Xuan Nga Cao, Juan Benjumea, Julien Karadayi, Mathieu Bernard, Laurent Besacier, Xavier Anguera, and Emmanuel Dupoux. 2017. The zero resource speech challenge 2017. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 323–330. IEEE. find
  12. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmentedse-quence data with recurrent neural networks. find
  13. ICML 2006 - Machine Learning, Proceedings of the Twenty-Third International Conference. find
  14. Awni Hannun, Ann Lee, Qiantong Xu, and Ronan Collobert. 2019. Sequence-to-sequence speech recognition with time-depth separable convolutions. CoRR, abs/1904.02619. — [arXiv:1904.02619](https://arxiv.org/abs/1904.02619) find
  15. He He, Jordan Boyd-Graber, and Hal Daumé III. 2016. Interpretese vs. translationese: The uniqueness of human strategies in simultaneous interpretation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  16. Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pages 187–197. find our card
  17. Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerdà, Javier Jorge, Nahuel Roselló, Adrià Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. 2020. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. In ICASSP 2020-2020 IEEE International Conference on Acoustic… find
  18. Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. 2019. Direct speech-to-speech translation with a sequence-to-sequence model. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 Sep… find
  19. Jacob Kahn, Ann Lee, and Awni Hannun. 2020a. Self-training for end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7084–7088. IEEE. find
  20. Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. 2020b. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conferenc… find
  21. Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux. 2020. Data augmenting contrastive learning of speech representations in the time domain. find
  22. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit, volume 5, pages 79–86. Citeseer. find
  23. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… find our card
  24. Ha Nguyen, Fethi Bougares, N. Tomashenko, Y. Estève, and L. Besacier. 2020. Investigating self-supervised pre-training for end-to-end speech translation. In INTERSPEECH. find
  25. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. find our card
  26. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. find our card
  27. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. IEEE. find
  28. J. Pino, Qiantong Xu, Xutai Ma, Mohammad Javad Dousti, and Yun Tang. 2020. Self-training for end-to-end speech translation. In INTERSPEECH. find
  29. Vineel Pratap, Awni Hannun, Qiantong Xu, Jeff Cai, Jacob Kahn, Gabriel Synnaeve, Vitaliy Liptchinsky, and Ronan Collobert. 2018. wav2letter++: The fastest open-source speech recognition system. CoRR, abs/1812.07625. find
  30. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. MLS: A Large-Scale Multilingual Dataset for Speech Research. In Proc. find our card
  31. Interspeech 2020, pages 2757– 2761. find
  32. Morgane Riviere and Emmanuel Dupoux. 2020. Towards unsupervised learning of speech features in the wild. find
  33. SLT 2020: IEEE Spoken Language Technology Workshop. find
  34. Morgane Rivière, Armand Joulin, Pierre-Emmanuel Mazaré, and Emmanuel Dupoux. 2020. Unsupervised pretraining transfers well across languages. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7414–7418. IEEE. find
  35. Thomas Schatz, Vijayaditya Peddinti, Francis Bach, Aren Jansen, Hynek Hermansky, and Emmanuel Dupoux. 2013. Evaluating speech features with the minimal-pair abx task: Analysis of the classical mfc/plp pipeline. In INTERSPEECH 2013: 14th Annual Conference of the International Speech Communication Ass… find
  36. Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020a. fairseq s2t: Fast speech-to-text modeling with fairseq. In Proceedings of the 2020 Conference of the Asian Chapter of the Association for Computational Linguistics (AACL): System Demonstrations. find
  37. Changhan Wang, Anne Wu, and Juan Pino. 2020b. Covost 2 and massively multilingual speech-to-text translation. arXiv e-prints, pages arXiv–2007. find
  38. Anne Wu, Changhan Wang, Juan Pino, and Jiatao Gu. 2020. Self-supervised representations improve end-to-end speech translation. In INTERSPEECH. find
  39. Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. 2020. Self-training and pre-training are complementary for speech recognition. arXiv preprint arXiv:2010.11430. — [arXiv:2010.11430](https://arxiv.org/abs/2010.11430) find
  40. H Scudder. 1965. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363–371. find
  41. Marcely Zanon Boito, William Havard, Mahault Garnerin, Éric Le Ferrand, and Laurent Besacier. 2020. MaSS: A large and clean multilingual corpus of sentence-aligned spoken utterances extracted from the Bible. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6486– 6493,… find
  42. Hiroaki Shimizu, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2014. Collection of a simultaneous translation corpus for comparative analysis. In LREC, pages 670–673. Citeseer. find
  43. Chen Zhang, Xu Tan, Yi Ren, Tao Qin, Kejun Zhang, and Tie-Yan Liu. 2020a. Uwspeech: Speech to speech translation for unwritten languages. arXiv preprint arXiv:2006.07926. — [arXiv:2006.07926](https://arxiv.org/abs/2006.07926) find
  44. Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised Pre-Training for Speech Recognition. In Proc. find
  45. Interspeech 2019, pages 3465–3469. find
  46. Vivek Kumar Rangarajan Sridhar, John Chen, and Srinivas Bangalore. 2013. Corpus analysis of simultaneous interpretation data for improving real time speech translation. In INTERSPEECH, pages 3468– 3472. find
  47. Yu Zhang, James Qin, Daniel S Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V Le, and Yonghui Wu. 2020b. Pushing the limits of semi-supervised learning for automatic speech recognition. arXiv preprint arXiv:2010.10504. — [arXiv:2010.10504](https://arxiv.org/abs/2010.10504) find
  48. Chaitanya Talnikar, Tatiana Likhomanenko, Ronan Collobert, and Gabriel Synnaeve. 2020. Joint masked cpc and ctc training for asr. arXiv preprint arXiv:2011.00093. — [arXiv:2011.00093](https://arxiv.org/abs/2011.00093) find
  49. Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. 2019. Speech-to-speech translation between untranscribed unknown languages. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 593–600. IEEE. find
  50. Hitomi Tohyama, Shigeki Matsubara, Koichiro Ryu, N Kawaguch, and Yasuyoshi Inagaki. 2004. Ciair simultaneous interpretation corpus. In Proc. Oriental COCOSDA. find
  51. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all find