Elamur Research
☰
home / papers / 2023 / SeamlessM4T: Massively Multilingual & Multimodal Machine Translation

SeamlessM4T: Massively Multilingual & Multimodal Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2308.11596

Summary

Text-based machine translation such as NLLB now covers more than 200 languages, but speech translation lags behind. Most speech-to-speech systems still chain separate ASR, T2TT and TTS components, favour high-resource languages and mainly translate into English. The Meta AI team presents SeamlessM4T, a single model that handles ASR, T2TT, S2TT, T2ST and S2ST for up to 100 languages.

They first pre-trained a w2v-BERT 2.0 speech encoder, without supervision, on roughly 1 million hours of open audio. They then mined SeamlessAlign, an automatically aligned speech–translation corpus of over 470,000 hours. A filtered subset was combined with human-labelled and pseudo-labelled material, giving 406,000 training hours.

SeamlessM4T-Large beats the previous best direct S2TT system on Fleurs and outperforms cascaded pipelines on several tasks. It also handles noise and speaker variation better and adds less toxicity than prior models. Models, inference code, Fairseq2 fine-tuning recipes and the SeamlessAlign metadata are openly released, which gives researchers a strong shared foundation for multilingual speech work.

Key points

  • Two released models: SeamlessM4T-Large (2.3B parameters) and SeamlessM4T-Medium (about 1.2B). Their coverage is:
    • S2ST: 100-eng and eng-35
    • S2TT: 100-eng and eng-95
    • ASR: 96 languages
    • T2TT: 95-eng and eng-95
    • zero-shot T2ST: 95-eng and eng-35
  • Speech translation results:
    • S2TT into English on Fleurs: +4.2 BLEU (~20%) over AudioPaLM-2-8B-AST.
    • S2TT from English on CoVoST 2: +2.8 BLEU over XLS-R-2B-S2T.
    • S2ST on Fleurs: +2.6 ASR-BLEU over 3-stage cascades.
    • S2ST on CVSS: +8.5 ASR-BLEU over Whisper-Large-v2 + YourTTS. The paper reports this as 50% in the introduction but 58% in the abstract.
  • ASR and text results: Fleurs ASR shows a 45% average WER reduction versus Whisper-Large-v2 across 77 shared languages. On Flores, T2TT matches NLLB-3.3B into English and gains about 1 chrF++ point from English.
  • Evaluation and safety:
    • New modality-agnostic metric Blaser 2.0.
    • Human XSTS scores above 4/5 for all 24 tested from-English languages.
    • Robustness gains over Whisper-Large-v2 of 38% for background noise and 49% for speaker variation.
    • Up to 63% less added toxicity than the prior state of the art.
  • Everything is open-sourced at github.com/facebookresearch/seamless_communication, built on the Fairseq2 toolkit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Topics: Machine translation

Cited by

168 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 162 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.11596 · arXiv 2308.11596

@misc{communication2023seamlessm4t,
  title = {{SeamlessM4T: Massively Multilingual \& Multimodal Machine Translation}},
  author = {Communication, Seamless and Barrault, Loic and Chung, Yu-An and Meglioli, Mariano Cora and Dale, David and Dong, Ning and Duquenne, Paul-Ambroise and Elsahar, Hady and Gong, Hongyu and Heffernan, Kevin and Hoffman, John and Klaiber, Christopher and Li, Pengwei and Licht, Daniel and Maillard, Jean and Rakotoarison, Alice and Sadagopan, Kaushik Ram and Wenzek, Guillaume and Ye, Ethan and Akula, Bapi and Chen, Peng-Jen and Hachem, Naji El and Ellis, Brian and Gonzalez, Gabriel Mejia and Haaheim, Justin and Hansanti, Prangthip and Howes, Russ and Huang, Bernie and Hwang, Min-Jae and Inaguma, Hirofumi and Jain, Somya and Kalbassi, Elahe and Kallet, Amanda and Kulikov, Ilia and Lam, Janice and Li, Daniel and Ma, Xutai and Mavlyutov, Ruslan and Peloquin, Benjamin and Ramadan, Mohamed and Ramakrishnan, Abinesh and Sun, Anna and Tran, Kevin and Tran, Tuan and Tufanov, Igor and Vogeti, Vish and Wood, Carleigh and Yang, Yilin and Yu, Bokai and Andrews, Pierre and Balioglu, Can and Costa-jussa, Marta R. and Celebi, Onur and Elbayad, Maha and Gao, Cynthia and Guzman, Francisco and Kao, Justine and Lee, Ann and Mourachko, Alexandre and Pino, Juan and Popuri, Sravya and Ropers, Christophe and Saleem, Safiyyah and Schwenk, Holger and Tomasello, Paden and Wang, Changhan and Wang, Jeff and Wang, Skyler},
  year = {2023},
  doi = {10.48550/arxiv.2308.11596},
  eprint = {2308.11596},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.11596}
}

References

193 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ACTFL. World-readiness standards for learning languages, 2023. URL https://www.actfl. org/educator-resources/world-readiness-standards-for-learning-languages. ↗
  2. Milind Agarwal, Sweta Agrawal, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Mingda Chen, William Chen, Khalid Choukri, Alexandra Chronopoulou, Anna Currey, Thierry Declerck, Qianqian Dong, Kevin Duh, Yannick Estève, Marcello F… find
  3. Antonios Anastasopoulos and David Chiang. Tied multitask learning for neural speech translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 82–91, New Orleans, Louisi… find
  4. Pierre Andrews, Guillaume Wenzek, Kevin Heffernan, Onur Çelebi, Anna Sun, Ammar Kamran, Yingzhe Guo, Alexandre Mourachko, Holger Schwenk, and Angela Fan. stopes -modular machine translation pipelines. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System D… find
  5. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… find our card
  6. Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondřej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang, Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Xing Shi, Sebastian Stüker, Marco Turch… find
  7. Mohamed Anwar, Bowen Shi, Vedanuj Goswami, Wei-Ning Hsu, Juan Pino, and Changhan Wang. Muavic: A multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation. arXiv preprint arXiv:2303.00628, 2023. — [arXiv:2303.00628](https://arxiv.org/abs/2303.00628) find
  8. Mikel Artetxe and Holger Schwenk. Margin-based parallel corpus mining with multilingual sentence embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3197–3203, Florence, Italy, July 2019a. Association for Computational Linguistics. doi: 10.18… find our card
  9. Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610, 2019b. doi: 10.1162/tacl\_a\_00288. URL https://aclanthology. org/Q19-1038. find our card
  10. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale. In Proc. Interspeech 2022… find our card
  11. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12449–12… find our card
  12. Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language, 2022. URL https://arxiv.org/abs/2202.03555. — [arXiv:2202.03555](https://arxiv.org/abs/2202.03555) ↗
  13. Parnia Bahar, Tobias Bieschke, and Hermann Ney. A comparative study on end-to-end speech to text translation. 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 792–799, 2019. URL https://api.semanticscholar.org/ CorpusID:204791999. ↗
  14. Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. mslam: Massively multilingual joint pre-training for speech and text. arXiv preprint arXiv:2202.01374, 2022. — [arXiv:2202.01374](https://arxiv.org/abs/2202.01374) find
  15. Aishwarya Belekar, Shivani Sunka, Neha Bhawar, and Sudhir Bagade. Voice based e-mail for the visually impaired. International Journal of Computer Applications, 175(16):8–12, 2020. find
  16. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021. find
  17. Luisa Bentivogli, Beatrice Savoldi, Matteo Negri, Mattia A. Di Gangi, Roldano Cattoni, and Marco Turchi. Gender in danger? evaluating speech translation technology on the MuST-SHE corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6923–6933, Onl… find
  18. Alexandre Berard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. Listen and translate: A proof of concept for end-to-end speech-to-text translation. In NeurIPS Workshop on End-to-end Learning for Speech and Audio Processing., 2016. find
  19. Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. End-to-end automatic speech translation of audiobooks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6224–6228. find
  20. Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: A language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language P… find our card
  21. Weicheng Cai, Danwei Cai, Shen Huang, and Ming Li. Utterance-level end-to-end language identification using attention-based cnn-blstm. In ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5991–5995. find
  22. Rich Caruana. Multitask learning. Machine learning, 28:41–75, 1997. find
  23. Edresson Casanova, Julian Weber, Christopher D Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning, pages 2709–2720. PMLR, 2022. find
  24. William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4960–4964, 2016. doi: 10.1109/ICASSP.2016.7472… — [doi:10.1109/icassp.2016.7472621](https://doi.org/10.1109/icassp.2016.7472621) find
  25. Mingda Chen, Paul-Ambroise Duquenne, Pierre Andrews, Justine Kao, Alexandre Mourachko, Holger Schwenk, and Marta R. Costa-jussà. BLASER: A text-free speech-to-speech translation evaluation metric. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: L… find
  26. Mingda Chen, Kevin Heffernan, Onur Çelebi, Alexandre Mourachko, and Holger Schwenk. xSIM++: An improved proxy to bitext mining performance for low-resource languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 101–109, T… find
  27. Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, Hirofumi Inaguma, Sravya Popuri, Changhan Wang, Juan Pino, Wei-Ning Hsu, and Ann Lee. Speech-to-speech translation for a real-world unwritten language. I… find
  28. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104–120. Springer, 2020. find our card
  29. Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro J. Moreno, Ankur Bapna, and Heiga Zen. Maestro: Matched speech text representations through modality matching. In Interspeech, 2022. URL https://api.semanticscholar.org/ CorpusID:248006130. ↗
  30. Yong Cheng, Yu Zhang, Melvin Johnson, Wolfgang Macherey, and Ankur Bapna. Mu 2 SLAM: Multitask, multilingual speech and language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conferenc… find
  31. Yuh-show Cheng, Elaine K Horwitz, and Diane L Schallert. Language anxiety: Differentiating writing and speaking components. Language learning, 49(3):417–446, 1999. find
  32. Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning, pages 3915–3924. PMLR, 2022. find
  33. Prafulla Kumar Choubey, Anna Currey, Prashant Mathur, and Georgiana Dinu. GFST: Gender-filtered self-training for more accurate gender in translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1640–1654, Online and Punta Cana, Dominican Republic… find
  34. Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. w2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244–250, 2021.… — [doi:10.1109/asru51503.2021](https://doi.org/10.1109/asru51503.2021) find
  35. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. arXiv preprint arXiv:2205.12446, 2022. URL https://arxiv.org/abs/2205.12446. ↗ our card
  36. Marta Costa-jussà, Christine Basta, Oriol Domingo, and André Rubungo. Occgen: Selection of real-world multilingual parallel data balanced in gender within occupations. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, vo… find
  37. Marta R. Costa-jussà, Christine Basta, and Gerard I. Gállego. Evaluating gender bias in speech translation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2141–2147, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthology.… ↗
  38. Marta R Costa-jussà, Pierre Andrews, Eric Smith, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Daniel Licht, and Carleigh Wood. Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale. arXiv preprint arXiv:2305.13198, 20… find
  39. Marta R. Costa-jussà, Carlos Escolano, Christine Basta, Javier Ferrando, Roser Batlle, and Ksenia Kharitonova. Interpreting gender bias in neural machine translation: Multilingual architecture matters. Proceedings of the AAAI Conference on Artificial Intelligence, 36 (11):11855–11863, Jun. 2022. doi… — [doi:10.1609/aaai.v36i11.21442](https://doi.org/10.1609/aaai.v36i11.21442) find
  40. Marta R. Costa-jussà, Eric Smith, Christophe Ropers, Daniel Licht, Jean Maillard, Javier Ferrando, and Carlos Escolano. Toxicity in multilingual machine translation at scale, 2023. find
  41. M.R. Costa-jussà. An analysis of gender bias studies in natural language processing. Nature Machine Intelligence, pages 495—-496, 2019. doi: 10.1038/s42256-019-0105-5. — [doi:10.1038/s42256-019-0105-5](https://doi.org/10.1038/s42256-019-0105-5) find
  42. Siddharth Dalmia, Brian Yan, Vikas Raunak, Florian Metze, and Shinji Watanabe. Searchable hidden intermediates for end-to-end models of decomposable sequence tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te… find
  43. Alexandre D’efossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. ArXiv, abs/2210.13438, 2022. URL https://api.semanticscholar. org/CorpusID:253097788. ↗ our card
  44. Najim Dehak, Pedro A Torres-Carrasquillo, Douglas Reynolds, and Reda Dehak. Language recognition via i-vectors and dimensionality reduction. In Twelfth annual conference of the international speech communication association, 2011. find
  45. Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. ECAPA-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In Helen Meng, Bo Xu, and Thomas Fang Zheng, editors, Interspeech 2020, 21st Annual Conference of the International Speech Communication… find
  46. ISCA, 2020. doi: 10.21437/Interspeech. 2020-2650. URL https://doi.org/10.21437/Interspeech.2020-2650. — [doi:10.21437/interspeech](https://doi.org/10.21437/interspeech) ↗
  47. Pranav Dheram, Murugesan Ramakrishnan, Anirudh Raju, I-Fan Chen, Brian King, Katherine Powell, Melissa Saboowala, Karan Shetty, and Andreas Stolcke. Toward fairness in speech recognition: Discovery and mitigation of performance disparities. arXiv preprint arXiv:2207.11345, 2022. — [arXiv:2207.11345](https://arxiv.org/abs/2207.11345) find
  48. Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. MuST-C: a Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a… find our card
  49. Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. Multimodal and multilingual embeddings for large-scale speech mining. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 15748–15761. Curr… find
  50. Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, and Holger Schwenk. T-modules: Translation modules for zero-shot cross-modal machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5794– 5806, Abu Dhabi, United Arab Emirates, December… find
  51. Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswami, Changhan Wang, Juan Pino, Benoît Sagot, and Holger Schwenk. SpeechMatrix: A large-scale mined corpus of multilingual speech-to-speech translations. In ACL (long paper), pages 16251–16269, 2023a. URL https://aclanth… find
  52. Paul-Ambroise Duquenne, Holger Schwenk, and Benoit Sagot. SONAR: sentence-level multimodal and language-agnostic representations, 2023b. URL https://arxiv.org/abs/ 2308.11466. ↗
  53. Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. Modular speech-to-text translation for zero-shot cross-modal transfer. In Interspeech, 2023c. find
  54. Peter Elbow. The shifting relationships between speech and writing. College composition and communication, 36(3):283–303, 1985. find
  55. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. Beyond english-centric multilingual ma… find
  56. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, Dublin, Ireland, May 2022. Association for Computa… find our card
  57. Sarith Fernando, Vidhyasaharan Sethu, Eliathamby Ambikairajah, and Julien Epps. Bidirectional modelling for short duration language identification. In Interspeech, pages 2809–2813, 2017. find
  58. Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681, 2021. — [arXiv:2111.12681](https://arxiv.org/abs/2111.12681) find
  59. Philip Gage. A new algorithm for data compression. C Users Journal, 12(2):23–38, 1994. find
  60. Marco Gaido, Matteo Negri, Mauro Cettolo, and Marco Turchi. Beyond voice activity detection: Hybrid audio segmentation for direct speech translation. In Mourad Abbas and Abed Alhakim Freihat, editors, 4th International Conference on Natural Language and Speech Processing, Trento, Italy, November 12-… find
  61. Gerard I. Gállego, Ioannis Tsiamas, Carlos Escolano, José A. R. Fonollosa, and Marta R. Costa-jussà. End-to-end speech translation with pre-trained models and adapters: UPC at IWSLT 2021. In Marcello Federico, Alex Waibel, Marta R. Costa-jussà, Jan Niehues, Sebastian Stüker, and Elizabeth Salesky, e… find
  62. Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems, 33:6616–6628, 2020. find
  63. Mattia Antonino Di Gangi, Matteo Negri, and Marco Turchi. One-to-many multilingual end-to-end speech translation. 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 585–592, 2019. URL https://api.semanticscholar.org/ CorpusID:203905407. ↗
  64. GBD 2019 Blindness and Vision Impairment Collaborators. Trends in prevalence of blindness and distance and near vision impairment over 30 years: an analysis for the global burden of disease study. The Lancet global health, 9(2):e130–e143, 2021. find
  65. Javier García Gilabert, Carlos Escolano, and Marta R. Costa-Jussà. Resetox: Re-learning attention weights for toxicity mitigation in machine translation, 2023. find
  66. Hongyu Gong, Ning Dong, Sravya Popuri, Vedanuj Goswami, Ann Lee, and Juan Pino. Multilingual speech-to-speech translation into multiple target languages. arXiv preprint arXiv:2307.08655, 2023. — [arXiv:2307.08655](https://arxiv.org/abs/2307.08655) find
  67. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational… find
  68. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33– 41, Sofia, Bulgaria, August 2013. Association for Compu… find
  69. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmented Transformer for Speech Recognition. In Proc. Interspeech 2020, pages 5036–5040, 2020. doi: 10.21437/Interspeech.2020-3015. find our card
  70. Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank. Glottolog database 4.6, 2022. find
  71. Caner Hazirbas, Joanna Bitton, Brian Dolhansky, Jacqueline Pan, Albert Gordo, and Cristian Canton Ferrer. Casual conversations: A dataset for measuring fairness in ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2289–2293, 2021. find
  72. Kevin Heffernan, Onur Çelebi, and Holger Schwenk. Bitext mining using distilled sentence representations for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2101–2112, Abu Dhabi, United Arab Emirates, December 2022. Association for Computationa… find
  73. Judith Holler and Stephen C Levinson. Multimodal language processing in human communication. Trends in Cognitive Sciences, 23(8):639–652, 2019. find
  74. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021. find our card
  75. Ke Hu, Tara N. Sainath, Ruoming Pang, and Rohit Prabhavalkar. Deliberation model based two-pass end-to-end speech recognition. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7799–7803, 2020. doi: 10.1109/ICASSP40776.2020.9053606. — [doi:10.1109/icassp40776.2020.9053606](https://doi.org/10.1109/icassp40776.2020.9053606) find
  76. Hirofumi Inaguma, Kevin Duh, Tatsuya Kawahara, and Shinji Watanabe. Multilingual end-to-end speech translation. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 570–577. find
  77. Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. UnitY: Two-pass direct speech-to-speech translation with discrete units. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis… find
  78. Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerda, Javier Jorge, Nahuel Roselló, Adria Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. Europarl-st: A multilingual corpus for speech translation of parliamentary debates. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech a… find
  79. Javier Iranzo-Sánchez, Jorge Civera, and Alfons Juan. From simultaneous to streaming machine translation by leveraging streaming history. arXiv preprint arXiv:2203.02459, 2022. find
  80. Anastasia Iskhakova, Daniyar Wolf, and Roman Meshcheryakov. Automated destructive behavior state detection on the 1d cnn-based voice analysis. In Speech and Computer: 22nd International Conference, SPECOM 2020, St. Petersburg, Russia, October 7–9, 2020, Proceedings, page 184–193, Berlin, Heidelberg,… — [doi:10.1007/978-3-030-60276-5\\\_19](https://doi.org/10.1007/978-3-030-60276-5\_19) find
  81. Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron J Weiss, Yuan Cao, Chung-Cheng Chiu, Naveen Ari, Stella Laurenzo, and Yonghui Wu. Leveraging weakly supervised data to improve end-to-end speech-to-text translation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Proce… find
  82. Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model. In Proc. Interspeech 2019, pages 1123–1127, 2019b. doi: 10.21437/Interspeech.2019-1951. — [doi:10.21437/interspeech.2019-1951](https://doi.org/10.21437/interspeech.2019-1951) find
  83. Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. Translatotron 2: High-quality direct speech-to-speech translation with voice preservation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International… find
  84. Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. CVSS corpus and massively multilingual speech-to-speech translation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6691–6703, Marseille, France, June 2022b. European Language Resources Association. U… find
  85. Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2019. find our card
  86. Youngjoo Jung and Bora Kim. Coexistence of multiple writing systems: Classifying digraphia in post-socialist countries. Journal of Eurasian Studies, page 18793665231188380, 2023. find
  87. Sameer Khurana, Antoine Laurent, and James Glass. Samu-xlsr: Semantically-aligned multimodal utterance-level cross-lingual speech representation. arXiv preprint arXiv:2205.08180, 2022. — [arXiv:2205.08180](https://arxiv.org/abs/2205.08180) find
  88. Svetlana Kiritchenko, Isar Nejadgholi, and Kathleen C Fraser. Confronting abusive language online: A survey from the ethical and human rights perspective. Journal of Artificial Intelligence Research, 71:431–478, 2021. find
  89. Tom Kocmi, Rachel Bawden, OndÅ™ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, Maja Popović,… find
  90. Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand, September 13-15 2005. URL https://aclanthology.org/2005.mtsummit-papers.11. ↗
  91. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14): 7684–7689, 2020. find
  92. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022–17033, 2020. find
  93. Robert Kraut, Jolene Galegher, Robert Fish, and Barbara Chalfonte. Task requirements and media choice in collaborative writing. Human–Computer Interaction, 7(4):375–407, 1992. find
  94. Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium, November 201… find our card
  95. Amit Kumar and Nicholas Epley. It’s surprisingly nice to hear you: Misunderstanding the impact of communication media can lead to suboptimal choices of how to connect with others. Journal of Experimental Psychology: General, 150(3):595, 2021. find
  96. Ravi Kumar Vuddagiri, Krishna Gurugubelli, Priyam Jain, Hari Krishna Vydana, and Anil Kumar Vuppala. IIITH-ILSC Speech Database for Indain Language Identification. In Proc. 6th Workshop on Spoken Language Technologies for Under-Resourced Languages (SLTU 2018), pages 56–60, 2018. doi: 10.21437/SLTU.2… — [doi:10.21437/sltu.2018-12](https://doi.org/10.21437/sltu.2018-12) find
  97. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336–… find
  98. Alon Lavie, Alexander H. Waibel, Lori S. Levin, Michael Finke, Donna Gates, Marsal Gavaldà, Torsten Zeppenfeld, and Puming Zhan. Janus-iii: speech-to-speech translation in multiple languages. 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, 1:99–102 vol.1, 1997. find
  99. Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. Direct speech-to-speech translation with discrete units. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volu… find
  100. Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. Textless speech-to-speech translation on real data. In Proceedings of the 2022 Conference of the North American Chapter of the Association for… find
  101. Sangmin-Michelle Lee. The effectiveness of machine translation in foreign language education: a systematic review and meta-analysis. Computer Assisted Language Learning, 36(1-2): 103–125, 2023. find
  102. Shahar Levy, Koren Lazar, and Gabriel Stanovsky. Collecting a large-scale gender bias dataset for coreference resolution and machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2470–2480, Punta Cana, Dominican Republic, November 2021. Association for… find
  103. M. Paul Lewis, editor. Ethnologue: Languages of the World. SIL International, Dallas, TX, USA, sixteenth edition, 2009. find
  104. Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. Multilingual speech translation from efficient finetuning of pretrained models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11… find
  105. Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. Multilingual speech translation from efficient finetuning of pretrained models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11… find
  106. Alvin M Liberman. The relation of speech to reading and writing. In Advances in psychology, volume 94, pages 167–178. Elsevier, 1992. find
  107. Daniel Licht, Cynthia Gao, Janice Lam, Francisco Guzman, Mona Diab, and Philipp Koehn. Consistent human evaluation of machine translation across language pairs. In Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages… find
  108. Chunxi Liu, Michael Picheny, Leda Sarı, Pooja Chitkara, Alex Xiao, Xiaohui Zhang, Mark Chou, Andres Alvarado, Caner Hazirbas, and Yatharth Saraf. Towards measuring fairness in speech recognition: Casual conversations dataset transcriptions. In ICASSP 2022-2022 IEEE International Conference on Acoust… find
  109. Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1871–1880, 2019. find
  110. Ignacio Lopez-Moreno, Javier Gonzalez-Dominguez, Oldrich Plchot, David Martinez, Joaquin Gonzalez-Rodriguez, and Pedro Moreno. Automatic language identification using deep neural networks. In 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5337–5341. find
  111. Alicia Lozano-Diez, Ruben Zazo-Candil, Javier Gonzalez-Dominguez, Doroteo T Toledano, and Joaquin Gonzalez-Rodriguez. An end-to-end approach to language identification in short utterances using convolutional neural networks. In Sixteenth Annual Conference of the International Speech Communication As… find
  112. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy, August 2019… find our card
  113. Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos. Attentive single-tasking of multiple tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1851–1860, 2019. find
  114. Katerina Markelova. Illiteracy: “another form of slavery”, Nov 2021. URL https://en. unesco.org/courier/2021-5/illiteracy-another-form-slavery. find
  115. Xiaoxiao Miao, Ian McLoughlin, and Yonghong Yan. A new time-frequency attention mechanism for tdnn and cnn-lstm-tdnn, with application to language identification. In Interspeech, pages 4080–4084, 2019. find
  116. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, page 220–229, New York, NY,… find
  117. Benjamin Muller, Belen Alastruey, Prangthip Hansanti, Elahe Kalbassi, Christophe Ropers, Eric Smith, Adina Williams, Luke Zettlemoyer, Pierre Andrews, and Marta R. Costa-jussà. Automatic pipeline for gender multilingual data characterisation at scale. arXiv preprint arXiv:, 2023. find
  118. S. Nakamura, K. Markov, H. Nakaiwa, G. Kikui, H. Kawai, T. Jitsuhiro, Jin-Song Zhang, H. Yamamoto, E. Sumita, and S. Yamamoto. The atr multilingual speech-to-speech translation system. Trans. Audio, Speech and Lang. Proc., 14(2):365–376, dec 2006. ISSN 1558-7916. doi: 10.1109/TSA.2005.860774. URL ht… — [doi:10.1109/tsa.2005.860774](https://doi.org/10.1109/tsa.2005.860774) find
  119. Mikel K Ngueajio and Gloria Washington. Hey asr system! why aren’t you more inclusive? automatic speech recognition systems’ bias and proposed bias mitigation techniques. a literature review. In International Conference on Human-Computer Interaction, pages 421–440. Springer, 2022. find
  120. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia-Gonzalez, Prangthip Hansanti, John Hof… find
  121. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations… find our card
  122. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computat… find our card
  123. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654– 7673, Online, November 2020. Association for Comput… find our card
  124. Juan Pino, Qiantong Xu, Xutai Ma, Mohammad Javad Dousti, and Yun Tang. Self-Training for End-to-End Speech Translation. In Proc. Interspeech 2020, pages 1476–1480, 2020. doi: 10.21437/Interspeech.2020-2938. — [doi:10.21437/interspeech.2020-2938](https://doi.org/10.21437/interspeech.2020-2938) find
  125. Ingo Plag, Christiane Dalton-Puffer, and Harald Baayen. Morphological productivity across speech and writing. English Language & Linguistics, 3(2):209–228, 1999. find
  126. Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech Resynthesis from Discrete Disentangled Self-Supervised Representations. In Proc. Interspeech 2021, 2021. find
  127. Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/W15-3049. URL https://aclanthology.org/W15-3… ↗ our card
  128. Bilal Porgali, Vítor Albiero, Jordan Ryda, Cristian Canton Ferrer, and Caner Hazirbas. The casual conversations v2 dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10–17, 2023. find
  129. Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Belgium, Brussels, October 2018. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/ W18-6319. ↗ our card
  130. Tomasz Potapczyk and Pawel Przybysz. Srpol’s system for the IWSLT 2020 end-to-end speech translation task. In Marcello Federico, Alex Waibel, Kevin Knight, Satoshi Nakamura, Hermann Ney, Jan Niehues, Sebastian Stüker, Dekai Wu, Joseph Mariani, and François Yvon, editors, Proceedings of the 17th Inte… find
  131. Tomasz Potapczyk and Pawel Przybysz. SRPOL’s system for the IWSLT 2020 end-to-end speech translation task. In Proceedings of the 17th International Conference on Spoken Language Translation, pages 89–94, Online, July 2020b. Association for Computational Linguistics. doi: 10.18653/v1/2020.iwslt-1.9.… find
  132. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Scaling speech technology to 1,000+ languages, 2023. find our card
  133. Marcelo Prates, Pedro Avelar, and Luís Lamb. Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications, 32, 05 2020. doi: 10.1007/s00521-019-04144-6. find
  134. Rebecca Qian, Candace Ross, Jude Fernandes, Eric Michael Smith, Douwe Kiela, and Adina Williams. Perturbation augmentation for fairer NLP. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9496–9521, Abu Dhabi, United Arab Emirates, December 2022. Assoc… find
  135. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356, 2022. find our card
  136. Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantad… find our card
  137. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong K… find
  138. Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online, November 2020. Association for Computational Linguistics. doi… find our card
  139. Adithya Renduchintala and Adina Williams. Investigating failures of automatic translationin the case of unambiguous gender. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3454–3469, Dublin, Ireland, May 2022. Association for… find
  140. Adithya Renduchintala, Denise Diaz, Kenneth Heafield, Xian Li, and Mona Diab. Gender bias amplification during speed-quality optimization in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confere… find
  141. Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Ryan Padfield, James Qin, Daniel Rozenberg, Tara N. Sainath, Johan Schalkwyk, Matthew Sharifi, Michelle… find our card
  142. Oleg Rybakov, Fadi Biadsy, Xia Zhang, Liyang Jiang, Phoenix Meadowlark, and Shivani Agrawal. Streaming parrotron for on-device speech-to-speech conversion. arXiv preprint arXiv:2210.13761, 2022. — [arXiv:2210.13761](https://arxiv.org/abs/2210.13761) find
  143. Tara N. Sainath, Ruoming Pang, David Rybach, Yanzhang He, Rohit Prabhavalkar, Wei Li, Mirkó Visontai, Qiao Liang, Trevor Strohman, Yonghui Wu, Ian McGraw, and Chung-Cheng Chiu. Two-Pass End-to-End Speech Recognition. In Proc. Interspeech 2019, pages 2773–2777, 2019. doi: 10.21437/Interspeech.2019-13… — [doi:10.21437/interspeech.2019-1341](https://doi.org/10.21437/interspeech.2019-1341) find
  144. Elizabeth Salesky, Matthew Wiesner, Jacob Bremerman, Roldano Cattoni, Matteo Negri, Marco Turchi, Douglas W. Oard, and Matt Post. The Multilingual TEDx Corpus for Speech Recognition and Translation. In Proc. Interspeech 2021, pages 3655–3659, 2021. doi: 10.21437/Interspeech.2021-11. — [doi:10.21437/interspeech.2021-11](https://doi.org/10.21437/interspeech.2021-11) find
  145. Juliana Schroeder, Michael Kardas, and Nicholas Epley. The humanizing voice: Speech reveals, and text conceals, a more thoughtful mind in the midst of disagreement. Psychological science, 28(12):1745–1762, 2017. find
  146. Holger Schwenk. Filtering and mining parallel data in a joint multilingual space. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 228–234, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18… find
  147. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. CCMatrix: Mining billions of high-quality parallel sentences on the web. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen… find our card
  148. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August 2016. Association for Computatio… find
  149. Khetam Al Sharou and Lucia Specia. A taxonomy and study of critical errors in machine translation. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 171–180, Ghent, Belgium, June 2022. European Association for Machine Translation. URL https://acl… find
  150. Peng Shen, Xugang Lu, Sheng Li, and Hisashi Kawai. Feature representation of short utterances based on knowledge distillation for spoken language identification. In Interspeech, pages 1813–1817, 2018. find
  151. Peng Shen, Xugang Lu, Sheng Li, and Hisashi Kawai. Interactive learning of teacher-student model for short utterance spoken language identification. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5981–5985. find
  152. Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio-visual speech representation by masked multimodal cluster prediction. arXiv preprint arXiv:2201.02184, 2022. — [arXiv:2201.02184](https://arxiv.org/abs/2201.02184) find
  153. Aditya Siddhant, Ankur Bapna, Orhan Firat, Yuan Cao, Mia Xu Chen, Isaac Caswell, and Xavier Garcia. Towards the next 1000 languages in multilingual machine translation: Exploring the synergy between supervised and self-supervised learning. CoRR, abs/2201.03110, 2022. URL https://arxiv.org/abs/2201.0… — [arXiv:2201.03110](https://arxiv.org/abs/2201.03110) ↗
  154. Silero. Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier. https://github.com/snakers4/silero-vad, 2021. ↗
  155. Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. “I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9180–9211, Ab… find
  156. David Snyder, Guoguo Chen, and Daniel Povey. Musan: A music, speech, and noise corpus. arXiv preprint arXiv:1510.08484, 2015. — [arXiv:1510.08484](https://arxiv.org/abs/1510.08484) find
  157. David Snyder, Daniel Garcia-Romero, Alan McCree, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. Spoken language recognition using x-vectors. In Odyssey, volume 2018, pages 105–111, 2018. find
  158. Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and André F. T. Martins. Findings of the WMT 2021 shared task on quality estimation. In Proceedings of the Sixth Conference on Machine Translation, pages 684–725, Online, November 2021. Association for Co… find our card
  159. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/… find our card
  160. Tzu-Wei Sung, Jun-You Liu, Hung-yi Lee, and Lin-shan Lee. Towards end-to-end speech-to-text translation with two-pass decoding. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7175–7179, 2019. doi: 10.1109/ICASSP.2019.8682801. — [doi:10.1109/icassp.2019.8682801](https://doi.org/10.1109/icassp.2019.8682801) find
  161. Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel. Improving speech translation by understanding and learning from the auxiliary text translation task. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference… find
  162. Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Speech-to-speech translation between untranscribed unknown languages. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 593–600. find
  163. Alexandre Trilla and Francesc Alias. Sentence-based sentiment analysis for expressive text-to-speech. IEEE transactions on audio, speech, and language processing, 21(2):223–233, 2012. find
  164. Ioannis Tsiamas, Gerard I. Gállego, José A. R. Fonollosa, and Marta R. Costa-jussà. SHAS: Approaching optimal Segmentation for End-to-End Speech Translation. In Proc. Interspeech 2022, pages 106–110, 2022. doi: 10.21437/Interspeech.2022-59. — [doi:10.21437/interspeech.2022-59](https://doi.org/10.21437/interspeech.2022-59) find
  165. Jörgen Valk and Tanel Alumäe. VoxLingua107: a dataset for spoken language recognition. In Proc. IEEE SLT Workshop, 2021. find
  166. Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(7):3614–3633, 2021. find
  167. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing… find
  168. Wolfgang Wahlster. Verbmobil: Foundations of speech-to-speech translation. In Artificial Intelligence, 2000. URL https://api.semanticscholar.org/CorpusID:30807920. ↗
  169. Li Wan, Prashant Sridhar, Yang Yu, Quan Wang, and Ignacio Lopez Moreno. Tuplemax loss for language identification. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5976–5980. find
  170. Changhan Wang, Juan Pino, Anne Wu, and Jiatao Gu. CoVoST: A diverse multilingual speech-to-text translation corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4197–4203, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-… find
  171. Changhan Wang, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Ann Lee, Peng-Jen Chen, Jiatao Gu, and Juan Pino. fairseq sˆ2: A scalable and integrable speech synthesis toolkit. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 143–152, Onl… find
  172. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Annual Meetin… find our card
  173. Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. CoVoST 2 and Massively Multilingual Speech Translation. In Proc. Interspeech 2021, pages 2247–2251, 2021c. doi: 10.21437/ Interspeech.2021-2027. find
  174. Chengyi Wang, Sanyuan Chen, Yu Wu, Zi-Hua Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers. ArXiv, abs/2301.02111, 2023a. URL https://api.semanticscholar.org/CorpusID… ↗ our card
  175. Skyler Wang, Ned Cooper, Margaret Eby, and Eun Seo Jo. From human-centered to social-centered artificial intelligence: Assessing chatgpt’s impact through disruptive events. arXiv preprint arXiv:2306.00227, 2023b. — [arXiv:2306.00227](https://arxiv.org/abs/2306.00227) find
  176. Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. VioLA: Unified codec language models for speech recognition, synthesis, and translation. May 2023c. find
  177. Yiming Wang, Jinyu Li, Heming Wang, Yao Qian, Chengyi Wang, and Yu Wu. Wav2vec-switch: Contrastive learning from original-noisy speech pairs for robust speech recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7097–7101. find
  178. Barack Wanjawa, Lilian Wanzare, Florence Indede, Owen McOnyango, Edward Ombui, and Lawrence Muchemi. Kencorpus: A kenyan language corpus of swahili, dholuo and luhya for natural language processing tasks. arXiv preprint arXiv:2208.12081, 2022. find
  179. Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Z. Chen. Sequence-to-sequence models can directly translate foreign speech. In Interspeech, 2017a. find
  180. Ron J Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. Sequence-to-sequence models can directly translate foreign speech. In Interspeech, 2017b. find
  181. Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. Deliberation networks: Sequence generation beyond one-pass decoding. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing S… find
  182. Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax. In Proceedings of the Twenty-Eighth International Joint Con… — [doi:10.24963/ijcai.2019/746](https://doi.org/10.24963/ijcai.2019/746) find
  183. Midia Yousefi and Dimitra Emmanouilidou. Audio-based toxic language classification using self-attentive convolutional neural network. In 29th European Signal Processing Conference, EUSIPCO 2021, Dublin, Ireland, August 23-27, 2021, pages 11–15. find
  184. IEEE, 2021. doi: 10.23919/EUSIPCO54536.2021.9616001. URL https://doi.org/10.23919/ EUSIPCO54536.2021.9616001. — [doi:10.23919/eusipco54536.2021.9616001](https://doi.org/10.23919/eusipco54536.2021.9616001) find
  185. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2022. URL https://api.semanticscholar. org/CorpusID:236149944. ↗ our card
  186. Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, Zhong Meng, Ke Hu, Andrew Rosenberg, Rohit Prabhavalkar, Daniel S. Park, Parisa Haghani, Jason Riesa, Ginger Perng, Hagen Soltau, Trevor Strohman, Bhuvana Ramabhadran, Tara Sainath,… find
  187. Zi-Hua Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. ArXiv, abs/2303.03926, 2023b. URL https://api.semantics… — [arXiv:2303.03926](https://arxiv.org/abs/2303.03926) ↗
  188. Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, and Furu Wei. SpeechUT: Bridging speech and text with hidden-unit for encoder-decoder based speech-text pre-training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1663–1676, Abu… find
  189. Ding Zhao, Tara N. Sainath, David Rybach, Pat Rondon, Deepti Bhatia, Bo Li, and Ruoming Pang. Shallow-Fusion End-to-End Contextual Biasing. In Proc. Interspeech 2019, pages 1418–1422, 2019. doi: 10.21437/Interspeech.2019-1209. — [doi:10.21437/interspeech.2019-1209](https://doi.org/10.21437/interspeech.2019-1209) find
  190. Jinming Zhao, Hao Yang, Gholamreza Haffari, and Ehsan Shareghi. M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation. In Proc. Interspeech 2022, pages 111–115, 2022. doi: 10.21437/Interspeech.2022-592. — [doi:10.21437/interspeech.2022-592](https://doi.org/10.21437/interspeech.2022-592) find
  191. Qiu-Shi Zhu, Jie Zhang, Zi-Qiang Zhang, Ming-Hui Wu, Xin Fang, and Li-Rong Dai. A noise-robust self-supervised pre-training model based speech representation learning for automatic speech recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASS… find
  192. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). UR… find our card
  193. Ethan Zuckerman. The polyglot internet, October 2008. URL https://ethanzuckerman. com/the-polyglot-internet/. find