Elamur Research
☰
home / papers / 2022 / FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech

FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2205.12446

Summary

Speech research lacked an evaluation set that covers many languages with aligned content. Most multilingual speech corpora span a few dozen languages at most, and almost none are parallel across all of them. FLEURS addresses this by adding human speech to the FLoRes-101 machine translation benchmark. The authors used the 2009 public FLoRes sentences, which come from English Wikipedia and were translated by humans. Native speakers read each sentence in 102 languages, and separate annotators checked the recordings. The release pairs the audio with raw, normalized and character-level transcripts. Every sentence keeps a shared index, so the corpus is n-way parallel. One resource therefore supports ASR, Speech LangID, speech translation and speech–text retrieval. Baselines come from fine-tuning two 600M-parameter pre-trained models: the speech-only w2v-bert-51 and the joint speech-text mSLAM. With about 12 hours of supervision per language, FLEURS gives low-resource and unseen languages a common testbed. Ukrainian is among the covered languages.

Key points

  • Scale and splits: 102 languages from 17 language families and 27 writing systems. Sentences are split 1509/150/350 into train/dev/test. Speech totals are about 987h train, 120h dev and 283h test.
  • Recording and validation: Up to three native-speaker recordings per sentence, sampled at 16kHz, with no augmentation. Speakers in test are disjoint from those in train/dev. About 21.5% of sentences have no validated recording in the first version.
  • Transcripts: Three versions are provided (SRC RAW, SRC NORM, SRC CHAR). A universal character vocabulary serves as both the modeling and the evaluation unit.
  • Baseline models: w2v-bert-51 (0.6B) was pre-trained on 429k hours of speech in 51 languages. mSLAM (0.6B) adds more than 10TiB of text in 101 languages. The FLEURS languages are divided into 54 seen and 48 unseen with respect to speech pre-training.
  • Retrieval results and availability: The reported Speech-Text retrieval baseline averages 76.9 (speech-to-text) and 74.4 (text-to-speech). CJK languages, Thai and Burmese score in single digits. The data is distributed through TFDS and Hugging Face as part of xtreme_s.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

201 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 187 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2205.12446 · arXiv 2205.12446

@misc{conneau2022fleurs,
  title = {{FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech}},
  author = {Conneau, Alexis and Ma, Min and Khanuja, Simran and Zhang, Yu and Axelrod, Vera and Dalmia, Siddharth and Riesa, Jason and Rivera, Clara and Bapna, Ankur},
  year = {2022},
  doi = {10.48550/arxiv.2205.12446},
  eprint = {2205.12446},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.12446}
}

References

26 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. of NIPS, 2017. [2] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Con-[15] A. W. Black, “CMU wil… find
  2. [16] J. Iranzo-Sánchez, J. A. Silvestre-Cerda, J. Jorge, N. Roselló, A. Giménez, A. Sanchis, J. Civera, and A. Juan, “Europarl-ST: A multilingual corpus for speech translation of parliamentary debates,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (… find
  3. [17] R. Cattoni, M. A. Di Gangi, L. Bentivogli, M. Negri, and M. Turchi, “Must-c: A multilingual corpus for end-to-end speech translation,” Computer Speech & Language, vol. 66, p. 101155, 2021. find
  4. [18] E. Salesky, M. Wiesner, J. Bremerman, R. Cattoni, M. Negri, M. Turchi, D. W. Oard, and M. Post, “The multilingual TEDx corpus for speech recognition and translation,” arXiv preprint arXiv:2102.01757, 2021. — [arXiv:2102.01757](https://arxiv.org/abs/2102.01757) find
  5. [19] Y. Jia, M. T. Ramanovich, Q. Wang, and H. Zen, “CVSS corpus and massively multilingual speech-to-speech translation,” arXiv preprint arXiv:2201.03713, 2022. find
  6. [20] N. Goyal, C. Gao, V. Chaudhary, P.-J. Chen, G. Wenzek, D. Ju, S. Krishnan, M. Ranzato, F. Guzman, and A. Fan, “The FLORES-101 evaluation benchmark for low-resource and multilingual machine translation,” arXiv preprint arXiv:2106.03193, 2021. find
  7. [21] A. Fan, S. Bhosale, H. Schwenk, Z. Ma, A. El-Kishky, S. Goyal, M. Baines, O. Celebi, G. Wenzek, V. Chaudhary et al., “Beyond english-centric multilingual machine translation,” Journal of Machine Learning Research, vol. 22, no. 107, pp. 1–48, 2021. find
  8. [22] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, “W2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” 2021. find
  9. [23] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” Proc. Interspeech 2019, pp. 2613–2617, 2019. find our card
  10. [24] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition.” in INTERSPEECH. ISCA, 2015, pp. 3586–3589. [Online]. Available: http://dblp. uni-trier.de/db/conf/interspeech/interspeech2015.html\#KoPPK15 find
  11. [25] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). find
  12. [26] Y. Zhang, D. S. Park, W. Han, J. Qin, A. Gulati, J. Shor, A. Jansen, Y. Xu, Y. Huang, S. Wang, Z. Zhou, B. Li, M. Ma, W. Chan, J. Yu, Y. Wang, L. Cao, K. C. Sim, B. Ramabhadran, T. N. Sainath, F. Beaufays, Z. Chen, Q. V. Le, C.-C. Chiu, R. Pang, and Y. Wu, “BigSSL: Exploring the frontier of lar… find
  13. [27] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mT5: A massively multilingual pre-trained text-to-text transformer,” arXiv preprint arXiv:2010.11934, 2020. find
  14. [28] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. find
  15. [29] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning, 2006, pp. 369–376. find
  16. [30] X. Li, S. Dalmia, J. Li, M. Lee, P. Littell, J. Yao, A. Anastasopoulos, D. R. Mortensen, G. Neubig, A. W. Black et al., “Universal phone recognition with a multilingual allophone system,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). find
  17. [31] M. Witbrock and A. G. Hauptmann, “Speech recognition and information retrieval: Experiments in retrieving spoken documents,” in Proceedings of the DARPA speech recognition workshop, vol. 97, 1997. find
  18. [32] S.-y. Ishikawa, T. Ikeda, K. Miki, F. Adachi, R. Isotani, K.-I. Iso, and A. Okumura, “Speech-activated text retrieval system for multimodal cellular phones,” in 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 1. find
  19. [33] L.-s. Lee, J. Glass, H.-y. Lee, and C.-a. Chan, “Spoken content retrieval—beyond cascading speech recognition with text retrieval,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 23, no. 9, pp. 1389–1420, 2015. find
  20. [34] A. Jain, M. Guo, K. Srinivasan, T. Chen, S. Kudugunta, C. Jia, Y. Yang, and J. Baldridge, “MURAL: Multimodal, multitask representations across languages,” in Findings of the Association for Computational Linguistics: EMNLP 2021. Punta Cana, Dominican Republic: Association for Computational Ling… find
  21. [35] P.-A. Duquenne, H. Gong, and H. Schwenk, “Multimodal and multilingual embeddings for large-scale speech mining,” Advances in Neural Information Processing Systems, vol. 34, 2021. find
  22. [36] Y. Yang, G. H. Ábrego, S. Yuan, M. Guo, Q. Shen, D. Cer, Y.-H. Sung, B. Strope, and R. Kurzweil, “Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax,” in IJCAI, 2019. find
  23. [37] F. Feng, Y. Yang, D. Cer, N. Arivazhagan, and W. Wang, “Language-agnostic BERT sentence embedding,” in Proceedings of ACL. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 878–891. [Online]. Available: https://aclanthology.org/2022.acl-long.62 ↗ our card
  24. [38] R. K. Mohapatra, T. K. Mishra, S. Panda, and B. Majhi, “Ohcs: A database for handwritten atomic odia character recognition,” in 2015 Fifth National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics (NCVPRIPG). find
  25. [39] D. van Esch, T. Lucassen, S. Ruder, I. Caswell, and C. E. Rivera, “Writing system and speaker metadata for 2,800+ language varieties,” in Proceedings of LREC, 2022. *Table 7: FLEURS full ASR results, reporting % CER (↓), for all 102 languages.* Western European Language ast bs ca hr da nl en fi… find
  26. Language it kea lb mt nb oc pt es sv cy hy be bg cs et w2v-bert-51 (0.6B) mSLAM (0.6B) 2.6 2.3 4.9 5.1 19.4 21.0 17.3 17.3 5.8 6.1 11.7 12.7 4.2 4.4 3.7 3.3 7.6 7.8 11.1 12.0 7.2 7.8 9.1 7.5 4.8 5.2 10.3 9.2 3.1 3.5 ka lv lt mk pl ro ru sr sk sl uk ar az he kk 30.7 31.0 4.4 4.5 12.8 11.6 11.8 9.8 5.… find