Elamur Research
☰
home / papers / 2023 / Scaling Speech Technology to 1,000+ Languages

Scaling Speech Technology to 1,000+ Languages

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2305.13516

Summary

Most speech technology today handles only about a hundred of the world's 7,000+ languages. This leaves most speakers without tools, and the gap may hasten language loss. The Massively Multilingual Speech (MMS) project from Meta AI widens coverage by roughly an order of magnitude or more. The authors collected recordings of the New Testament read aloud in many languages, from sources such as Faith Comes By Hearing. They paired the audio with text using a scalable forced-alignment pipeline, refined it over two rounds, and filtered out weak segments through cross-validated monolingual ASR. They also gathered unlabeled audio for thousands more languages. Self-supervised wav2vec 2.0 pre-training then allowed them to fine-tune models for speech recognition, speech synthesis, and language identification from only modest labeled data per language. On 54 FLEURS languages, the single multilingual ASR model cuts Whisper's word error rate by more than half while using far less labeled data. All models are publicly released. The authors also examine religious-domain bias and find it slight.

Key points

  • Datasets: MMS-lab (paired speech–text, 1,107 languages, 44.7K hours), MMS-lab-U (unlabeled, 1,362 languages, 55K hours), and MMS-unlab (unlabeled, 3,809 languages, 7.7K hours).
  • Models: wav2vec 2.0 pre-trained on 1,406 languages, one ASR model and TTS models for 1,107 languages, and a language identification (LID) model covering 4,017 languages.
  • Alignment pipeline: a first aligner trained on 8K hours in 127 languages, then a second aligner covering 1,130 languages and 31K hours, followed by quality filtering.
  • Main result: more than twice lower word error rate than Whisper on 54 FLEURS languages.
  • Bias analysis: MMS ASR output contains religion-skewed words only 0.7% (absolute) more often than models trained on FLEURS. The code and models are in fairseq (examples/mms).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

217 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 208 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.13516 · arXiv 2305.13516

@misc{pratap2023scaling,
  title = {{Scaling Speech Technology to 1,000+ Languages}},
  author = {Pratap, Vineel and Tjandra, Andros and Shi, Bowen and Tomasello, Paden and Babu, Arun and Kundu, Sayani and Elkahky, Ali and Ni, Zhaoheng and Vyas, Apoorv and Fazel-Zarandi, Maryam and Baevski, Alexei and Adi, Yossi and Zhang, Xiaohui and Hsu, Wei-Ning and Conneau, Alexis and Auli, Michael},
  year = {2023},
  doi = {10.48550/arxiv.2305.13516},
  eprint = {2305.13516},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.13516}
}

References

105 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber. Common voice: A massively-multilingual speech corpus. Proc. of LREC, 2020. find our card
  2. J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv, 2016. find our card
  3. A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli. XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale. In Proc. Interspeech 2022, pages 2278–2282, 2022. doi: 10.21437/Interspeech.2022-… find our card
  4. A. Baevski, S. Schneider, and M. Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In Proc. of ICLR, 2020a. find
  5. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Proc. of NeurIPS, 2020b. find our card
  6. A. Baevski, W.-N. Hsu, A. Conneau, and M. Auli. Unsupervised speech recognition. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. find
  7. M. Baines, S. Bhosale, V. Caggiano, N. Goyal, S. Goyal, M. Ott, B. Lefaudeux, V. Liptchinsky, M. Rabbat, S. Sheiffer, A. Sridhar, and M. Xu. Fairscale: A general purpose modular pytorch library for high performance and large scale training. https://github.com/facebookresearch/fairscale, 2021. ↗
  8. A. Bapna, I. Caswell, J. Kreutzer, O. Firat, D. van Esch, A. Siddhant, M. Niu, P. Baljekar, X. Garcia, W. Macherey, T. Breiner, V. Axelrod, J. Riesa, Y. Cao, M. X. Chen, K. Macherey, M. Krikun, P. Wang, A. Gutkin, A. Shah, Y. Huang, Z. Chen, Y. Wu, and M. Hughes. Building machine translation systems… — [arXiv:2205.03983](https://arxiv.org/abs/2205.03983) find
  9. A. Bapna, C. Cherry, Y. Zhang, Y. Jia, M. Johnson, Y. Cheng, S. Khanuja, J. Riesa, and A. Conneau. mslam: Massively multilingual joint pre-training for speech and text. ArXiv, abs/2202.01374, 2022b. — [arXiv:2202.01374](https://arxiv.org/abs/2202.01374) find
  10. A. W. Black. Cmu wilderness multilingual speech dataset. In Proc. of ICASSP, 2019. find
  11. H. Bourlard, J. Dines, et al. Current trends in multilingual speech processing. Sadhana, 2011. find
  12. L. Bromham, R. Dinnage, H. Skirgard, A. Ritchie, M. Cardillo, F. Meakins, S. Greenhill, and X. Hua. Global predictors of language endangerment and the future of linguistic diversity. Nature Ecology and Evolution, 2021. find
  13. L. Burget, P. Schwarz, et al. Multilingual acoustic modeling for speech recognition based on subspace gaussian mixture models. In Proc. of ICASSP, 2010. find
  14. X. Cai, J. Yuan, Y. Bian, G. Xun, J. Huang, and K. Church. W-CTC: a connectionist temporal classification loss with wild cards. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=0RqDp8FCW5Z. ↗
  15. E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. Gölge, and M. A. Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning, pages 2709–2720. PMLR, 2022. find
  16. W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals. Listen, attend and spell. arXiv, 2015. find
  17. W. Chan, D. Park, C. Lee, Y. Zhang, Q. Le, and M. Norouzi. Speechstew: Simply mix all available speech recognition data to train one large neural network. arXiv, abs/2104.02133, 2021. — [arXiv:2104.02133](https://arxiv.org/abs/2104.02133) find
  18. T. Chen, B. Xu, C. Zhang, and C. Guestrin. Training deep nets with sublinear memory cost. arXiv, abs/1604.06174, 2016. find
  19. W. Chen, B. Yan, J. Shi, Y. Peng, S. Maiti, and S. Watanabe. Improving massively multilingual asr with auxiliary ctc objectives. In Proc. of ICASSP, 2023. find
  20. Z. Chen, Y. Zhang, A. Rosenberg, B. Ramabhadran, P. Moreno, A. Bapna, and H. Zen. Maestro: Matched speech text representations through modality matching. arXiv, abs/2204.03409, 2022. — [arXiv:2204.03409](https://arxiv.org/abs/2204.03409) find
  21. J. Cho, M. K. Baskar, et al. Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling. In Proc. of IEEE SLT, 2018. find
  22. C. Christodouloupoulos and M. Steedman. A massively parallel corpus: the bible in 100 languages. In Proc. of LREC, 2015. find our card
  23. R. Collobert, C. Puhrsch, and G. Synnaeve. Wav2letter: an end-to-end convnet-based speech recognition system. arXiv, abs/1609.03193, 2016. — [arXiv:1609.03193](https://arxiv.org/abs/1609.03193) find
  24. A. Conneau and G. Lample. Cross-lingual language model pretraining. Proc. of NeurIPS, 2019. find
  25. A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv, abs/2006.13979, 2020a. — [arXiv:2006.13979](https://arxiv.org/abs/2006.13979) find
  26. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proc. of ACL, 2020b. find
  27. A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. arXiv, 2022. find our card
  28. A. Defossez, G. Synnaeve, and Y. Adi. Real time speech enhancement in the waveform domain. In Interspeech, 2020. find
  29. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proc. of NAACL, 2019. find
  30. S. Dieleman, A. van den Oord, and K. Simonyan. The challenge of realistic music generation: modelling raw audio at scale. Proc of NIPS, 2018. find
  31. L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using real NVP. In ICLR. OpenReview.net, 2017. find
  32. D. Doukhan, E. Lechapt, M. Evrard, and J. Carrive. Ina’s mirex 2018 music and speech detection system. In Proc. of MIREX, 2018. find
  33. Z. Fan, M. Li, S. Zhou, and B. Xu. Exploring wav2vec 2.0 on speaker verification and language identification. arXiv, 2021. find
  34. M. J. F. Gales, K. M. Knill, A. Ragni, and S. P. Rath. Speech recognition and keyword spotting for low-resource languages: Babel project research at cued. In Spoken Language Technologies for Under-Resourced Languages, 2014. find
  35. A. Graves, S. Fernández, and F. Gomez. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proc. of ICML, 2006. find
  36. M. He, J. Yang, L. He, and F. K. Soong. Multilingual byte2speech models for scalable low-resource speech synthesis. arXiv, abs/arXiv:2103.03541, 2021. — [arXiv:2103.03541](https://arxiv.org/abs/2103.03541) find
  37. K. Heafield. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. find our card
  38. G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, and J. Dean. Multilingual acoustic models using distributed deep neural networks. In Proc. of ICASSP, 2013. find
  39. U. Hermjakob, J. May, and K. Knight. Out-of-the-box universal Romanization tool uroman. In Proceedings of ACL 2018, System Demonstrations, pages 13–18, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-4003. URL https: //aclanthology.org/P18-4003. find
  40. J. L. Hieronymus. Ascii phonetic symbols for the world’s languages: Worldbet. JIPA, 23:72, 1993. find
  41. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019. find
  42. W.-N. Hsu, A. Sriram, A. Baevski, T. Likhomanenko, Q. Xu, V. Pratap, J. Kahn, A. Lee, R. Collobert, G. Synnaeve, et al. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. arXiv preprint arXiv:2104.01027, 2021. — [arXiv:2104.01027](https://arxiv.org/abs/2104.01027) find
  43. K. Ito and L. Johnson. The lj speech dataset. https://keithito.com/LJ-Speech-Dataset/, 2017. ↗
  44. E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. Proc. of ICLR, 2016. find our card
  45. T. Javed, S. Doddapaneni, A. Raman, K. S. Bhogale, G. Ramesh, A. Kunchukuttan, P. Kumar, and M. M. Khapra. Towards building asr systems for the next billion users. In Proc. of AAAI CAI, 2022. find
  46. H. Jegou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell., 33(1):117–128, Jan. 2011. find
  47. F. Jia, N. R. Koluguri, J. Balam, and B. Ginsburg. Ambernet: A compact end-to-end model for spoken language identification. arXiv, abs/2210.15781, 2022. — [arXiv:2210.15781](https://arxiv.org/abs/2210.15781) find
  48. J. D. Kahn, V. Pratap, T. Likhomanenko, Q. Xu, A. Hannun, J. Cai, P. Tomasello, A. Lee, E. Grave, G. Avidov, et al. Flashlight: Enabling innovation in tools for machine learning. In International Conference on Machine Learning, pages 10557–10574. PMLR, 2022. find
  49. A. Kannan, A. Datta, et al. Large-scale multilingual speech recognition with a streaming end-to-end model. In Proc. of Interspeech, 2019. find
  50. J. Kim, J. Kong, and J. Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In Proc. of ICML, 2021. find
  51. D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In Proc. of ICLR, 2015. find
  52. D. P. Kingma and M. Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013. find our card
  53. A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Mengesha, C. Toups, J. R. Rickford, D. Jurafsky, and S. Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 2020. find
  54. J. Kong, J. Kim, and J. Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In NeurIPS, 2020. find
  55. L. Kürzinger, D. Winkelbauer, L. Li, T. Watzel, and G. Rigoll. Ctc-segmentation of large corpora for german end-to-end speech recognition. In A. Karpov and R. Potapova, editors, Speech and Computer, pages 267–278, Cham, 2020. Springer International Publishing. find
  56. C. Leong, J. Nemecek, J. Mansdorfer, A. Filighera, A. Owodunni, and D. Whitenack. Bloom library: Multimodal datasets in 300+ languages for a variety of downstream tasks. In Proc. of EMNLP, 2022. find
  57. M. P. Lewis, G. F. Simon, and C. D. Fennig. Ethnologue: Languages of the world, nineteenth edition. Online version: http://www.ethnologue.com, 2016. ↗
  58. B. Li, Y. Zhang, T. Sainath, Y. Wu, and W. Chan. Bytes are all you need: End-to-end multilingual speech recognition and synthesis with bytes. In Proc. of ICASSP, 2019. find
  59. B. Li, R. Pang, T. N. Sainath, A. Gulati, Y. Zhang, J. Qin, P. Haghani, W. R. Huang, M. Ma, and J. Bai. Scaling end-to-end models for large-scale multilingual asr. In Proc. of ASRU, 2021. find
  60. X. Li, F. Metze, D. R. Mortensen, A. W. Black, and S. Watanabe. Asr2k: Speech recognition for around 2000 languages without audio. arXiv preprint arXiv:2209.02842, 2022. — [arXiv:2209.02842](https://arxiv.org/abs/2209.02842) find
  61. T. Likhomanenko, G. Synnaeve, and R. Collobert. Who needs words? lexicon-free speech recognition. In Proc. of Interspeech, 2019. find
  62. T. Likhomanenko, Q. Xu, V. Pratap, P. Tomasello, J. Kahn, G. Avidov, R. Collobert, and G. Synnaeve. Rethinking evaluation in ASR: are our models robust enough? In Proc. of Interspeech, 2020. find
  63. H. Lin, L. Deng, D. Yu, Y.-f. Gong, A. Acero, and C.-H. Lee. A study on multilingual acoustic modeling for large vocabulary asr. In Proc. of ICASSP, 2009. find
  64. A. H. Liu, W.-N. Hsu, M. Auli, and A. Baevski. Towards end-to-end unsupervised speech recognition. arXiv, 2022. find
  65. Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv, abs/2001.08210, 2020. find
  66. L. Lugosch, T. Likhomanenko, G. Synnaeve, and R. Collobert. Pseudo-labeling for massively multilingual speech recognition. In Proc. of ICASSP, 2022. find
  67. M-AILABS. The m-ailabs speech dataset, 2018. URL https://www.caito.de/2019/01/03/the-m-ailabs-speech-dataset/. ↗
  68. M. Mauch and S. Dixon. Pyin: A fundamental frequency estimator using probabilistic threshold distributions. 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 659–663, 2014. find
  69. A. McCarthy, R. Wicks, D. Lewis, A. Mueller, W. Wu, O. Adams, G. Nicolai, M. Post, and D. Yarowsky. The johns hopkins university bible corpus: 1600+ tongues for typological exploration. In Proc. of LREC, 2020. find
  70. J. Meyer, D. I. Adelani, E. Casanova, A. Öktem, D. W. J. Weber, S. Kabongo, E. Salesky, I. Orife, C. Leong, P. Ogayo, C. Emezue, J. Mukiibi, S. Osei, A. Agbolo, V. Akinode, B. Opoku, S. Olanrewaju, J. Alabi, and S. Muhammad. Bibletts: a large, high-fidelity, multilingual, and uniquely african speech… — [arXiv:2207.03546](https://arxiv.org/abs/2207.03546) find
  71. T. Nekvinda and O. Dušek. One model, many languages: Meta-learning for multilingual text-to-speech. In Interspeech, 2020. find
  72. NLLB\_Team, M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. M. Gonzalez, P. Hansanti, J. Hoffman, S. Jarrett, K. R. Sadagopan, D. Rowe, S. Spruit, C. Tran, P… find
  73. M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL System Demonstrations, 2019. find our card
  74. K. Park and J. Kim. g2pe. https://github.com/Kyubyong/g2p, 2019. ↗
  75. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely. The kaldi speech recognition toolkit. In Proc. of ASRU, 2011. find
  76. V. Pratap, A. Hannun, Q. Xu, J. Cai, J. Kahn, G. Synnaeve, V. Liptchinsky, and R. Collobert. Wav2letter++: A fast open-source speech recognition system. In Proc. of ICASSP, 2019. find
  77. V. Pratap, A. Sriram, P. Tomasello, A. Hannun, V. Liptchinsky, G. Synnaeve, and R. Collobert. Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters. In Proc. of Interspeech 2020, 2020a. find
  78. V. Pratap, A. Sriram, et al. Massively multilingual asr: 50 languages, 1 model, 1 billion parameters. arXiv, abs/2007.03001, 2020b. — [arXiv:2007.03001](https://arxiv.org/abs/2007.03001) find
  79. V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert. Mls: A large-scale multilingual dataset for speech research. In Proc. of Interspeech, 2020c. find our card
  80. V. Pratap, A. Hannun, G. Synnaeve, and R. Collobert. Star temporal classification: Sequence modeling with partially labeled data. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=ldRyJb\_cjXa. ↗
  81. T. Qian, K. Hollingshead, S.-y. Yoon, K.-y. Kim, and R. Sproat. A python toolkit for universal transliteration. In Proc. of LREC, 2010. find
  82. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. arXiv, 2022. find our card
  83. S. Rajbhandari, O. Ruwase, J. Rasley, S. Smith, and Y. He. Zero-infinity: Breaking the GPU memory wall for extreme scale deep learning. arXiv, abs/2104.07857, 2021. find
  84. M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio. SpeechBrain: A general-purpose speech toolkit, 2021.… — [arXiv:2106.04624](https://arxiv.org/abs/2106.04624) find
  85. F. Ribeiro, D. Florêncio, C. Zhang, and M. Seltzer. Crowdmos: An approach for crowdsourcing mean opinion score studies. In 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 2416–2419. find
  86. T. Saeki, S. Maiti, X. Li, S. Watanabe, S. Takamichi, and H. Saruwatari. Learning to speak from text: Zero-shot multilingual text-to-speech with unsupervised text pretraining. arXiv, abs/2301.12596, 2023a. — [arXiv:2301.12596](https://arxiv.org/abs/2301.12596) find
  87. T. Saeki, H. Zen, Z. Chen, N. Morioka, G. Wang, Y. Zhang, A. Bapna, A. Rosenberg, and B. Ramabhadran. Virtuoso: Massive multilingual speech-text joint semi-supervised learning for text-to-speech. In Proc. of ICASSP, 2023b. find
  88. S. Schneider, A. Baevski, R. Collobert, and M. Auli. wav2vec: Unsupervised pre-training for speech recognition. In Proc. of Interspeech, 2019. find
  89. M. Staib, T. H. Teh, A. Torresquintero, D. S. R. Mohan, L. Foglianti, R. Lenain, and J. Gao. Phonological features for 0-shot multilingual speech synthesis. In Interspeech, 2020. find
  90. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. Proc. of NIPS, 2014. find
  91. X. Tan, T. Qin, F. Soong, and T.-Y. Liu. A survey on neural speech synthesis. arXiv preprint arXiv:2106.15561, 2021. — [arXiv:2106.15561](https://arxiv.org/abs/2106.15561) find
  92. A. Tjandra, D. G. Choudhury, F. Zhang, K. Singh, A. Conneau, A. Baevski, A. Sela, Y. Saraf, and M. Auli. Improved language identification through cross-lingual self-supervised learning. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6877–… find
  93. A. Tjandra, N. Singhal, D. Zhang, O. Kalinli, A. Mohamed, D. Le, and M. L. Seltzer. Massively multilingual asr on 70 languages: Tokenization, architecture, and generalization capabilities. arXiv, abs/2211.05756, 2022b. — [arXiv:2211.05756](https://arxiv.org/abs/2211.05756) find
  94. S. Toshniwal, T. N. Sainath, R. J. Weiss, B. Li, P. Moreno, E. Weinstein, and K. Rao. Multilingual speech recognition with a single end-to-end model. In Proc. of ICASSP, 2018. find
  95. J. Valk and T. Alumäe. Voxlingua107: a dataset for spoken language recognition. In Proc. of SLT, 2020. find
  96. A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. Proc. of NIPS, 2018. find our card
  97. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Proc. of NIPS, 2017. find
  98. C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proc. of ACL, 2021. find our card
  99. S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai. ESPnet: End-to-end speech processing toolkit. In Proceedings of Interspeech, pages 2207–2211, 2018. doi: 10.21437/Interspeech.2018-1456. URL… — [doi:10.21437/interspeech.2018-1456](https://doi.org/10.21437/interspeech.2018-1456) find
  100. J. Wells. Computer-coding the ipa: a proposed extension of sampa, 1995. URL https://www.phon. ucl.ac.uk/home/sampa/x-sampa.htm. ↗
  101. Y.-Y. Yang, M. Hira, Z. Ni, A. Chourdia, A. Astafurov, C. Chen, C.-F. Yeh, C. Puhrsch, D. Pollack, D. Genzel, D. Greenberg, E. Z. Yang, J. Lian, J. Mahadeokar, J. Hwang, J. Chen, P. Goldsborough, P. Roy, S. Narenthiran, S. Watanabe, S. Chintala, V. Quenneville-Bélair, and Y. Shi. Torchaudio: Buildin… find
  102. S.-Y. Yoon, K.-Y. Kim, and R. Sproat. Multilingual transliteration using feature based phonetic method. In Proc. of ACL, 2007. find
  103. Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Z. Chen, R. Skerry-Ryan, Y. Jia, A. Rosenberg, and B. Ramabhadran. Learning to speak fluently in a foreign language: Multilingual speech synthesis and cross-language voice cloning. In Interspeech, 2019. find
  104. Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V. Axelrod, G. Wang, Z. Meng, K. Hu, A. Rosenberg, R. Prabhavalkar, D. S. Park, P. Haghani, J. Riesa, G. Perng, H. Soltau, T. Strohman, B. Ramabhadran, T. Sainath, P. Moreno, C.-C. Chiu, J. Schalkwyk, F. Beaufays, and Y. Wu. Googl… find
  105. Z. Zhang, L. Zhou, C. Wang, S. Chen, Y. Wu, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv, abs/2303.03926, 2023b. — [arXiv:2303.03926](https://arxiv.org/abs/2303.03926) find