Elamur Research
☰
home / papers / 2023 / Seamless: Multilingual Expressive and Streaming Speech Translation

Seamless: Multilingual Expressive and Streaming Speech Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2312.05187

Summary

Machine speech translation has broadened its language coverage, but it still sounds unlike human conversation. It discards the speaker's voice and prosody, and it typically waits for a full utterance before producing output. This Meta FAIR work tackles both gaps with a family of open models. The base is SeamlessM4T v2, which adds a new w2v-BERT 2.0 encoder, the UnitY2 architecture, and extra automatically aligned data for low-resource languages. Two models are built on top of it. SeamlessExpressive transfers vocal style and prosodic features, including speech rate and pauses. SeamlessStreaming uses the EMMA attention mechanism to translate simultaneously across many language pairs. Their key parts are then combined into Seamless, presented as the first publicly available system for real-time expressive cross-lingual speech translation. The authors also introduce new automatic and human evaluation protocols and a responsible-AI toolkit. Models, code and a watermark detector are released openly, which makes this a practical foundation for multilingual speech projects, including those involving Ukrainian.

Key points

  • SeamlessAlign data: The expanded corpus adds 114,800 hours of automatically aligned speech across 76 languages. SeamlessM4T v2 accepts speech or text input in nearly 100 languages.
  • SeamlessM4T v2 architecture: Pre-training for the w2v-BERT 2.0 encoder used 4.5M hours of audio without labels. UnitY2 predicts speech units with a non-autoregressive unit decoder and hierarchical upsampling, which uses data more efficiently.
  • The two derived models:
    • SeamlessExpressive translates to and from English across five languages while keeping voice style, rhythm and tone.
    • SeamlessStreaming, built on Efficient Monotonic Multihead Attention (EMMA), handles many-to-many simultaneous ASR, S2TT and S2ST.
  • Evaluation:
    • New automatic metrics: AutoPCP and a rhythm evaluation toolkit.
    • Human evaluation: XSTS, MOS and a modified PCP.
    • Latency: Ending Offset and an adapted Average Lagging.
  • Responsible AI:
    • What the authors describe as the first red-teaming effort for multimodal MT.
    • Detection and mitigation of added toxicity.
    • Systematic gender-bias analysis.
    • Inaudible localized watermarking (SeamlessWM) against deepfakes.
    • All releases are at github.com/facebookresearch/seamless_communication.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

130 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.05187 · arXiv 2312.05187

@misc{communication2023seamless,
  title = {{Seamless: Multilingual Expressive and Streaming Speech Translation}},
  author = {Communication, Seamless and Barrault, Loic and Chung, Yu-An and Meglioli, Mariano Coria and Dale, David and Dong, Ning and Duppenthaler, Mark and Duquenne, Paul-Ambroise and Ellis, Brian and Elsahar, Hady and Haaheim, Justin and Hoffman, John and Hwang, Min-Jae and Inaguma, Hirofumi and Klaiber, Christopher and Kulikov, Ilia and Li, Pengwei and Licht, Daniel and Maillard, Jean and Mavlyutov, Ruslan and Rakotoarison, Alice and Sadagopan, Kaushik Ram and Ramakrishnan, Abinesh and Tran, Tuan and Wenzek, Guillaume and Yang, Yilin and Ye, Ethan and Evtimov, Ivan and Fernandez, Pierre and Gao, Cynthia and Hansanti, Prangthip and Kalbassi, Elahe and Kallet, Amanda and Kozhevnikov, Artyom and Gonzalez, Gabriel Mejia and Roman, Robin San and Touret, Christophe and Wong, Corinne and Wood, Carleigh and Yu, Bokai and Andrews, Pierre and Balioglu, Can and Chen, Peng-Jen and Costa-jussa, Marta R. and Elbayad, Maha and Gong, Hongyu and Guzman, Francisco and Heffernan, Kevin and Jain, Somya and Kao, Justine and Lee, Ann and Ma, Xutai and Mourachko, Alex and Peloquin, Benjamin and Pino, Juan and Popuri, Sravya and Ropers, Christophe and Saleem, Safiyyah and Schwenk, Holger and Sun, Anna and Tomasello, Paden and Wang, Changhan and Wang, Jeff and Wang, Skyler and Williamson, Mary},
  year = {2023},
  doi = {10.48550/arxiv.2312.05187},
  eprint = {2312.05187},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.05187}
}

References

171 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chinese ai governance rules, 2023. URL http://www.cac.gov.cn/2023-07/13/c\_1690898327029107.htm. Accessed on October 18, 2023. ↗
  2. European ai act, 2023. URL https://artificialintelligenceact.eu/. Accessed on October 18, 2023. ↗
  3. Ensuring safe, secure, and trustworthy ai, 2023. URL https://www.whitehouse.gov/wp-content/uploads/2023/07/ Ensuring-Safe-Secure-and-Trustworthy-AI.pdf. Accessed on October 18, 2023. ↗
  4. M. Agarwal, S. Agrawal, A. Anastasopoulos, L. Bentivogli, O. Bojar, C. Borg, M. Carpuat, R. Cattoni, M. Cettolo, M. Chen, W. Chen, K. Choukri, A. Chronopoulou, A. Currey, T. Declerck, Q. Dong, K. Duh, Y. Estève, M. Federico, S. Gahbiche, B. Haddow, B. Hsu, P. Mon Htut, H. Inaguma, D. Javorský, J.… find
  5. P. Aguero, J. Adell, and A. Bonafonte. Prosody generation for speech-to-speech translation. In 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, volume 1, pages I–I. find
  6. A. Anastasopoulos, O. Bojar, J. Bremerman, R. Cattoni, M. Elbayad, M. Federico, X. Ma, S. Nakamura, M. Negri, J. Niehues, J. Pino, E. Salesky, S. Stüker, K. Sudoh, M. Turchi, A. Waibel, C. Wang, and M. Wiesner. FINDINGS OF THE IWSLT 2021 EVALUATION CAMPAIGN. In M. Federico, A. Waibel, M. R. Costa-j… find
  7. A. Anastasopoulos, L. Barrault, L. Bentivogli, M. Zanon Boito, O. Bojar, R. Cattoni, A. Currey, G. Dinu, K. Duh, M. Elbayad, C. Emmanuel, Y. Estève, M. Federico, C. Federmann, S. Gahbiche, H. Gong, R. Grundkiewicz, B. Haddow, B. Hsu, D. Javorský, V. Kloudová, S. Lakew, X. Ma, P. Mathur, P. McName… find
  8. P. Andrews, G. Wenzek, K. Heffernan, O. Çelebi, A. Sun, A. Kamran, Y. Guo, A. Mourachko, H. Schwenk, and A. Fan. stopes - modular machine translation pipelines. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 258–265, Abu Dhabi… find
  9. E. Ansari, A. Axelrod, N. Bach, O. Bojar, R. Cattoni, F. Dalvi, N. Durrani, M. Federico, C. Federmann, J. Gu, F. Huang, K. Knight, X. Ma, A. Nagesh, M. Negri, J. Niehues, J. Pino, E. Salesky, X. Shi, S. Stüker, M. Turchi, A. Waibel, and C. Wang. FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN. In M.… find
  10. G. K. Anumanchipalli, L. C. Oliveira, and A. W. Black. Intent transfer in speech-to-speech machine translation. In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 153–158. find
  11. N. Arivazhagan, C. Cherry, W. Macherey, C.-C. Chiu, S. Yavuz, R. Pang, W. Li, and C. Raffel. Monotonic Infinite Lookback Attention for Simultaneous Machine Translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1313–1323, Florence, Italy, July… find
  12. M. Artetxe and H. Schwenk. Margin-based parallel corpus mining with multilingual sentence embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3197–3203, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P… find our card
  13. J. E. Avila and N. G. Ward. Towards cross-language prosody transfer for dialog, 2023. find
  14. L. J. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. CoRR, abs/1607.06450, 2016. find our card
  15. A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli. XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale. In Proc. Interspeech 2022, pages 2278–2282, 2022. doi: 10.21437/Interspeech.2022-… find our card
  16. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12449–12460. Curran Associates,… find our card
  17. L. Bentivogli, M. Cettolo, M. Gaido, A. Karakanta, A. Martinelli, M. Negri, and M. Turchi. Cascade versus direct speech translation: Do the differences still make a difference? arXiv preprint arXiv:2106.01045, 2021. find
  18. C. M. Bishop. Mixture density networks. 1994. find
  19. Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour. Audiolm: A language modeling approach to audio generation. IEEE ACM Trans. Audio Speech Lang. Process., 31:2523–2533, 2023. URL https://doi.org/10.1109/TA… ↗ our card
  20. E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. Gölge, and M. A. Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning, pages 2709–2720. PMLR, 2022. find
  21. G. Chen, Y. Wu, S. Liu, T. Liu, X. Du, and F. Wei. Wavmark: Watermarking for audio generation. arXiv preprint arXiv:2308.12770, 2023a. — [arXiv:2308.12770](https://arxiv.org/abs/2308.12770) find
  22. M. Chen, P.-A. Duquenne, P. Andrews, J. Kao, A. Mourachko, H. Schwenk, and M. R. Costa-jussà. BLASER: A text-free speech-to-speech translation evaluation metric. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9064–9079, Toro… find
  23. P.-J. Chen, K. Tran, Y. Yang, J. Du, J. Kao, Y.-A. Chung, P. Tomasello, P.-A. Duquenne, H. Schwenk, H. Gong, H. Inaguma, S. Popuri, C. Wang, J. Pino, W.-N. Hsu, and A. Lee. Speech-to-speech translation for a real-world unwritten language. In Findings of the Association for Computational Linguistics:… find
  24. S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE J. Sel. Top. Signal Process., 16(6):1505–1518,… find
  25. C.-C. Chiu* and C. Raffel*. Monotonic chunkwise attention. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=Hko85plCW. ↗
  26. C.-C. Chiu, J. Qin, Y. Zhang, J. Yu, and Y. Wu. Self-supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning, pages 3915–3924. PMLR, 2022. find
  27. K. Cho and M. Esipova. Can neural machine translation do simultaneous translation? arXiv preprint arXiv:1606.02012, 2016. cjadams, J. Sorensen, J. Elliott, L. Dixon, M. McDonald, nithum, and W. Cukierski. Toxic comment classification challenge, 2017. URL https://kaggle.com/competitions/jigsaw-toxic-… — [arXiv:1606.02012](https://arxiv.org/abs/1606.02012) ↗
  28. D. Cokely. The effects of lag time on interpreter errors. Sign Language Studies, pages 341–375, 1986. find
  29. A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006.13979, 2020. — [arXiv:2006.13979](https://arxiv.org/abs/2006.13979) find
  30. A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. arXiv preprint arXiv:2205.12446, 2022. URL https://arxiv.org/abs/2205.12446. ↗ our card
  31. M. R. Costa-jussà, P. Andrews, E. Smith, P. Hansanti, C. Ropers, E. Kalbassi, C. Gao, D. Licht, and C. Wood. Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale. In EMNLP, 2023. find
  32. M. Costa-jussà. An analysis of gender bias studies in natural language processing. Nature Machine Intelligence, pages 495—-496, 2019. doi: 10.1038/s42256-019-0105-5. — [doi:10.1038/s42256-019-0105-5](https://doi.org/10.1038/s42256-019-0105-5) find
  33. M. R. Costa-jussà, C. Escolano, C. Basta, J. Ferrando, R. Batlle, and K. Kharitonova. Interpreting gender bias in neural machine translation: Multilingual architecture matters. Proceedings of the AAAI Conference on Artificial Intelligence, 36(11):11855–11863, Jun. 2022. doi: 10.1609/aaai.v36i11.214… — [doi:10.1609/aaai.v36i11.21442](https://doi.org/10.1609/aaai.v36i11.21442) find
  34. M. R. Costa-jussà, D. Dale, M. Elbayad, and B. Yu. Added toxicity mitigation at inference time for massively multilingual and multimodal translation. CoRR, abs/2311.06532, 2023a. URL https://arxiv.org/abs/2311.06532. — [arXiv:2311.06532](https://arxiv.org/abs/2311.06532) ↗
  35. M. R. Costa-jussà, E. Smith, C. Ropers, D. Licht, J. Maillard, J. Ferrando, and C. Escolano. Toxicity in multilingual machine translation at scale. In EMNLP, 2023b. find
  36. J. Costello. Aac intervention in the intensive care unit: The children’s hospital boston model. Augmentative and alternative communication, 16(3):137–153, 2000. find
  37. F. Cummins, F. Gers, and J. Schmidhuber. Comparing prosody across many languages. Technical report, I.D.S.I.A, 1999. find
  38. D. Dale, E. Voita, L. Barrault, and M. R. Costa-jussà. Detecting and mitigating hallucinations in machine translation: Model internal workings alone do well, sentence similarity Even better. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P… find
  39. F. Dalvi, N. Durrani, H. Sajjad, and S. Vogel. Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volu… find
  40. A. Défossez, J. Copet, G. Synnaeve, and Y. Adi. High fidelity neural audio compression. ArXiv, abs/2210.13438, 2022. URL https://api.semanticscholar.org/CorpusID:253097788. ↗ our card
  41. L. Delander, M. Hammarstedt, J. MÅnsson, and E. Nyberg. Integration of immigrants: The role of language proficiency and experience. Evaluation review, 29(1):24–41, 2005. doi: 10.1177/0193841X04270230. URL https: //doi.org/10.1177/0193841X04270230. — [doi:10.1177/0193841x04270230](https://doi.org/10.1177/0193841x04270230) find
  42. B. Desplanques, J. Thienpondt, and K. Demuynck. ECAPA-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In H. Meng, B. Xu, and T. F. Zheng, editors, Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtu… find
  43. ISCA, 2020. doi: 10.21437/Interspeech.2020-2650. URL https: //doi.org/10.21437/Interspeech.2020-2650. — [doi:10.21437/interspeech.2020-2650](https://doi.org/10.21437/interspeech.2020-2650) find
  44. H. Ding and L. Hargraves. Stress-associated poor health among adult immigrants with a language barrier in the united states. Journal of immigrant and minority health, 11:446–452, 2009. find
  45. Q. T. Do, S. Sakti, and S. Nakamura. Toward expressive speech translation: A unified sequence-to-sequence lstms approach for translating words and emphasis. In INTERSPEECH, pages 2640–2644, 2017. find
  46. Q. Dong, Z. Huang, Q. Tian, C. Xu, T. Ko, Y. Zhao, S. Feng, T. Li, K. Wang, X. Cheng, F. Yue, Y. Bai, X. Chen, L. Lu, Z. Ma, Y. Wang, M. Wang, and Y. Wang. Polyvoice: Language models for speech to speech translation. CoRR, abs/2306.02982, 2023. URL https://doi.org/10.48550/arXiv.2306.02982. — [doi:10.48550/arxiv.2306.02982](https://doi.org/10.48550/arxiv.2306.02982) ↗
  47. Z.-Y. Dou and G. Neubig. Word alignment by fine-tuning embeddings on parallel corpora. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2021. find
  48. Z. Du, B. Sisman, K. Zhou, and H. Li. Expressive voice conversion: A joint framework for speaker identity and emotional style transfer. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 594–601. find
  49. P.-A. Duquenne, H. Gong, and H. Schwenk. Multimodal and multilingual embeddings for large-scale speech mining. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 15748–15761. Curran Associates, Inc., 20… find
  50. P.-A. Duquenne, K. Heffernan, A. Mourachko, B. Sagot, and H. Schwenk. Expressivity-aware SONAR: speech decoder, 2023a. find
  51. P.-A. Duquenne, H. Schwenk, and B. Sagot. SONAR: sentence-level multimodal and language-agnostic representations, 2023b. URL https://arxiv.org/abs/2308.11466. ↗
  52. J. Escudé Font and M. R. Costa-jussà. Equalizing gender bias in neural machine translation with word embeddings techniques. In M. R. Costa-jussà, C. Hardmeier, W. Radford, and K. Webster, editors, Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 147–154, Flor… find
  53. F. Eyben, M. Wöllmer, and S. Schuller, Björn. "opensmile - the munich versatile and fast open-source audio feature extractor". In Proceedings of the 2010 ACM Multimedia (MM), ACM, Florence, Italy, pages 1459–1462, 2010. find
  54. P. Fernandez, G. Couairon, H. Jégou, M. Douze, and T. Furon. The stable signature: Rooting watermarks in latent diffusion models. ICCV, 2023. find
  55. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In T. Cohn, Y. He, and Y. Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online, Nov. 2020. Association for… find our card
  56. S. Ghosh, S. Lepcha, S. Sakshi, R. R. Shah, and S. Umesh. Detoxy: A large-scale multimodal dataset for toxicity classification in spoken utterances. In Interspeech, 2021. URL https://api.semanticscholar.org/CorpusID: 247940269. ↗
  57. H. Gong, N. Dong, S. Popuri, V. Goswami, A. Lee, and J. Pino. Multilingual speech-to-speech translation into multiple target languages. arXiv preprint arXiv:2307.08655, 2023a. — [arXiv:2307.08655](https://arxiv.org/abs/2307.08655) find
  58. S. Gong, M. Li, J. Feng, Z. Wu, and L. Kong. DiffuSeq: Sequence to sequence text generation with diffusion models. In International Conference on Learning Representations, ICLR, 2023b. find
  59. J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281, 2017a. find our card
  60. J. Gu, G. Neubig, K. Cho, and V. O. Li. Learning to Translate in Real-time with Neural Machine Translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1053–1062, Valencia, Spain, Apr. 2017b. Association… find
  61. A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang. Conformer: Convolution-augmented Transformer for Speech Recognition. In Proc. Interspeech 2020, pages 5036–5040, 2020. doi: 10.21437/Interspeech.2020-3015. find our card
  62. H. Hammarström, R. Forkel, M. Haspelmath, and S. Bank. Glottolog database 4.6, 2022. find
  63. L. Hanu and Unitary team. Detoxify. Github. https://github.com/unitaryai/detoxify, 2020. ↗
  64. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  65. W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021. find our card
  66. W.-C. Huang, B. Peloquin, J. Kao, C. Wang, H. Gong, E. Salesky, Y. Adi, A. Lee, and P.-J. Chen. A holistic cascade system, benchmark, and human evaluation protocol for expressive speech-to-speech translation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processin… find
  67. J. Hutchins. Multiple uses of machine translation and computerised translation tools. Machine Translation, pages 13–20, 2009. find
  68. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 448–456. PMLR, 2015. find
  69. ITU-T Recommendation P.808. Subjective evaluation of speech quality with a crowdsourcing approach, 2018. find
  70. Y. Jadoul, B. Thompson, and B. de Boer. Introducing Parselmouth: A Python interface to Praat. Journal of Phonetics, 71:1–15, 2018. doi: https://doi.org/10.1016/j.wocn.2018.07.001. — [doi:10.1016/j.wocn.2018.07.001](https://doi.org/10.1016/j.wocn.2018.07.001) ↗
  71. Y. Jia, M. Johnson, W. Macherey, R. J. Weiss, Y. Cao, C.-C. Chiu, N. Ari, S. Laurenzo, and Y. Wu. Leveraging weakly supervised data to improve end-to-end speech-to-text translation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7180–7184… find
  72. Y. Jia, M. T. Ramanovich, T. Remez, and R. Pomerantz. Translatotron 2: High-quality direct speech-to-speech translation with voice preservation. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning,… find
  73. Y. Jia, M. Tadmor Ramanovich, Q. Wang, and H. Zen. CVSS corpus and massively multilingual speech-to-speech translation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6691–6703, Marseille, France, June 2022b. European Language Resources Association. URL https://… find
  74. J. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3): 535–547, 2019. find our card
  75. L. Juvela and X. Wang. Collaborative watermarking for adversarial speech synthesis. arXiv preprint arXiv:2309.15224, 2023. — [arXiv:2309.15224](https://arxiv.org/abs/2309.15224) find
  76. E. Kharitonov, A. Lee, A. Polyak, Y. Adi, J. Copet, K. Lakhotia, T. A. Nguyen, M. Rivière, A. Mohamed, E. Dupoux, and W. Hsu. Text-free prosody-aware generative spoken language modeling. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Associatio… find
  77. E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. ArXiv, abs/2302.03540, 2023. URL https://api.semanticscholar.org/CorpusID:256627687. ↗
  78. J. Kim, S. Kim, J. Kong, and S. Yoon. Glow-TTS: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33:8067–8077, 2020. find
  79. J. Kim, J. Kong, and J. Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages 5530–5540. PMLR, 2021. find
  80. J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.10226, 2023. find our card
  81. A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Mengesha, C. Toups, J. R. Rickford, D. Jurafsky, and S. Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117 (14):7684–7689, 2020. find
  82. J. Kong, J. Kim, and J. Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022–17033, 2020. find
  83. R. Kraut, J. Galegher, R. Fish, and B. Chalfonte. Task requirements and media choice in collaborative writing. Human–Computer Interaction, 7(4):375–407, 1992. find
  84. T. Kudo and J. Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium, Nov. 2018. Assoc… find our card
  85. M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint arXiv:2306.15687, 2023. — [arXiv:2306.15687](https://arxiv.org/abs/2306.15687) find
  86. A. Lee, P.-J. Chen, C. Wang, J. Gu, S. Popuri, X. Ma, A. Polyak, Y. Adi, Q. He, Y. Tang, J. Pino, and W.-N. Hsu. Direct speech-to-speech translation with discrete units. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3327–333… find
  87. A. Lee, H. Gong, P.-A. Duquenne, H. Schwenk, P.-J. Chen, C. Wang, S. Popuri, Y. Adi, J. Pino, J. Gu, and W.-N. Hsu. Textless speech-to-speech translation on real data. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… find
  88. J. Lee and S. Watanabe. Intermediate loss regularization for CTC-based speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6224–6228. find
  89. J. Lee, E. Mansimov, and K. Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1173–1182, Brussels, Belg… find our card
  90. K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini. Deduplicating training data makes language models better. CoRR, 2021. URL https://arxiv.org/abs/2107.06499. ↗
  91. S. C. Levinson. Turn-taking in human communication–origins and implications for language processing. Trends in cognitive sciences, 20(1):6–14, 2016. find
  92. M. P. Lewis, editor. Ethnologue: Languages of the World. SIL International, Dallas, TX, USA, sixteenth edition, 2009. find
  93. D. Licht, C. Gao, J. Lam, F. Guzman, M. Diab, and P. Koehn. Consistent human evaluation of machine translation across language pairs. In Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 309–321, Orlando, USA, Sep… find
  94. C. Liu, J. Zhang, H. Fang, Z. Ma, W. Zhang, and N. Yu. Dear: A deep-learning-based audio re-recording resilient watermarking. In B. Williams, Y. Chen, and J. Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of… find
  95. AAAI Press, 2023. doi: 10.1609/aaai.v37i11.26550. URL https: //doi.org/10.1609/aaai.v37i11.26550. — [doi:10.1609/aaai.v37i11.26550](https://doi.org/10.1609/aaai.v37i11.26550) find
  96. Z. Liu and B. Mak. Cross-lingual multi-speaker text-to-speech synthesis for voice cloning without using parallel corpus for unseen speakers. arXiv preprint arXiv:1911.11601, 2019. — [arXiv:1911.11601](https://arxiv.org/abs/1911.11601) find
  97. K. Lueck and M. Wilson. Acculturative stress in latino immigrants: The impact of social, socio-psychological and migration-related factors. International Journal of Intercultural Relations, 35(2):186–195, 2011. find
  98. M. Ma, L. Huang, H. Xiong, R. Zheng, K. Liu, B. Zheng, C. Zhang, Z. He, H. Liu, X. Li, H. Wu, and H. Wang. STACL: Simultaneous Translation with Implicit Anticipation and Controllable Latency using Prefix-to-Prefix Framework. In Proceedings of the 57th Annual Meeting of the Association for Computatio… find our card
  99. X. Ma, C. Zhou, X. Li, G. Neubig, and E. Hovy. FlowSeq: Non-autoregressive conditional sequence generation with generative flow. In K. Inui, J. Jiang, V. Ng, and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Co… find
  100. X. Ma, M. J. Dousti, C. Wang, J. Gu, and J. Pino. SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 144–150, Online, Oct. 2020a. Association for Computational Linguist… find
  101. X. Ma, J. Pino, and P. Koehn. SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Nat… find
  102. X. Ma, J. M. Pino, J. Cross, L. Puzon, and J. Gu. Monotonic multihead attention. In International Conference on Learning Representations, 2020c. URL https://openreview.net/forum?id=Hyg96gBKPS. ↗
  103. X. Ma, Y. Wang, M. J. Dousti, P. Koehn, and J. Pino. Streaming simultaneous speech translation with augmented memory transformer. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7523–7527, 2021. doi: 10.1109/ICASSP39728.2021.9414897. — [doi:10.1109/icassp39728.2021.9414897](https://doi.org/10.1109/icassp39728.2021.9414897) find
  104. X. Ma, A. Sun, S. Ouyang, H. Inaguma, and P. Tomasello. Efficient monotonic multihead attention. AI at Meta, Nov. 2023. URL https://ai.meta.com/research/publications/efficient-monotonic-multihead-attention/. ↗
  105. M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pages 498–502, 2017. find
  106. M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, page 220–229, New York, NY, USA, 2019. Association for Computing M… find
  107. M. Morise, F. Yokomori, and K. Ozawa. World: A vocoder-based high-quality speech synthesis system for real-time applications. IEICE Trans. Inf. Syst., 99-D:1877–1884, 2016. find
  108. S. Nakamura. Overcoming the language barrier with speech translation technology. Technical report, Citeseer, 2009. find
  109. G. Neubig, S. Sakti, T. Toda, S. Nakamura, et al. Collection and analysis of a japanese-english emphasized speech corpora. In 2014 17th Oriental Chapter of the International Committee for the Co-ordination and Standardization of Speech Databases and Assessment Techniques (COCOSDA), pages 1–5. find
  110. M. K. Ngueajio and G. Washington. Hey asr system! why aren’t you more inclusive? automatic speech recognition systems’ bias and proposed bias mitigation techniques. a literature review. In International Conference on Human-Computer Interaction, pages 421–440. Springer, 2022. find
  111. T. A. Nguyen, W.-N. Hsu, A. D’Avirro, B. Shi, I. Gat, M. Fazel-Zarani, T. Remez, J. Copet, G. Synnaeve, M. Hassid, F. Kreuk, Y. Adi, and E. Dupoux. Expresso: A benchmark and analysis of discrete expressive speech resynthesis, 2023. find
  112. T. Q. Nguyen and D. Chiang. Transfer learning across low-resource, related languages for neural machine translation. arXiv preprint arXiv:1708.09803, 2017. find our card
  113. NLLB Team, M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. Mejia-Gonzalez, P. Hansanti, J. Hoffman, S. Jarrett, K. R. Sadagopan, D. Rowe, S. Spruit, C. Tr… find
  114. M. F. Orellana, L. Dorner, and L. Pulido. Accessing assets: Immigrant youth’s work as family translators or" para-phrasers". Social problems, 50(4):505–524, 2003. find
  115. B. N. Oreshkin, P. Rodriguez, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 719–729, Red Hook, NY, USA, 2018. Curran Associates Inc. find
  116. S. Papi, M. Gaido, M. Negri, and M. Turchi. Over-generation cannot be rewarded: Length-adaptive average lagging for simultaneous speech translation. In J. Ive and R. Zhang, editors, Proceedings of the Third Workshop on Automatic Simultaneous Translation, pages 12–17, Online, July 2022. Association f… find
  117. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguis… find our card
  118. D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. In Proc. Interspeech 2019, pages 2613–2617, 2019. find our card
  119. J. Parry, D. Palaz, G. Clarke, P. Lecomte, R. Mead, M. Berger, and G. Hofer. Analysis of deep learning architectures for cross-corpus speech emotion recognition. In Interspeech, pages 1656–1660, 2019. find
  120. E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. C. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018. find our card
  121. E. Perez, S. Huang, H. F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. CoRR, abs/2202.03286, 2022. URL https://arxiv.org/abs/2202.03286. ↗ our card
  122. J. Pino, Q. Xu, X. Ma, M. J. Dousti, and Y. Tang. Self-Training for End-to-End Speech Translation. In Proc. Interspeech 2020, pages 1476–1480, 2020. doi: 10.21437/Interspeech.2020-2938. — [doi:10.21437/interspeech.2020-2938](https://doi.org/10.21437/interspeech.2020-2938) find
  123. A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux. Speech Resynthesis from Discrete Disentangled Self-Supervised Representations. In Proc. Interspeech 2021, 2021. find
  124. M. Popović. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal, Sept. 2015. Association for Computational Linguistics. doi: 10.18653/v1/W15-3049. URL https://aclanthology.org/W15-3049. ↗ our card
  125. K. Prajwal, R. Mukhopadhyay, V. P. Namboodiri, and C. Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 484–492, 2020. find
  126. V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi, A. Baevski, Y. Adi, X. Zhang, W.-N. Hsu, A. Conneau, and M. Auli. Scaling speech technology to 1,000+ languages, 2023. find our card
  127. L. Qian, H. Zhou, Y. Bao, M. Wang, L. Qiu, W. Zhang, Y. Yu, and L. Li. Glancing transformer for non-autoregressive neural machine translation. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Inte… find
  128. L. Qu, T. Li, C. Weber, T. Pekarek-Rosin, F. Ren, and S. Wermter. Disentangling prosody representations with unsupervised speech reconstruction. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:39–54, 2022. find
  129. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356, 2022. find our card
  130. C. Raffel, M.-T. Luong, P. J. Liu, R. J. Weiss, and D. Eck. Online and linear-time attention by enforcing monotonic alignments. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pages 2837–2846, Sydney, NSW, Australia, Aug. 2017. JMLR.org. find
  131. H. Rashkin, E. M. Smith, M. Li, and Y.-L. Boureau. Towards empathetic open-domain conversation models: a new benchmark and dataset. In ACL, 2019. find
  132. Reed, 2020. facebook-xi-jinping-mr-shithole". find
  133. URL https://www.theguardian.com/technology/2020/jan/18/M. Reid, V. J. Hellendoorn, and G. Neubig. Diffuser: Discrete diffusion via edit-based reconstruction. arXiv preprint arXiv:2210.16886, 2022. — [arXiv:2210.16886](https://arxiv.org/abs/2210.16886) ↗
  134. Y. Ren, J. Liu, X. Tan, C. Zhang, T. Qin, Z. Zhao, and T.-Y. Liu. SimulSpeech: End-to-End Simultaneous Speech to Text Translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3787–3796, Online, July 2020. Association for Computational Linguistics… find
  135. Y. Ren, C. Hu, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu. FastSpeech 2: Fast and high-quality end-to-end text-to-speech. In Proc. ICLR, 2021a. find
  136. Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu. FastSpeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2021b. URL https://openreview.net/ forum?id=piLPYqxtWuA. ↗
  137. S. Rouard, F. Massa, and A. Défossez. Hybrid transformers for music source separation. In ICASSP 23, 2023. find
  138. P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. de Chaumont Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, H. Muckenhirn, D. Padfield, J. Qin, D. Rozenberg, T. N. Sainath, J. Schalkwyk, M. Sharifi, M. T. Ramanovich, M. Tagliasacchi, A. Tudor, M. Velimirovic, D. Vinc… find our card
  139. P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. de Chaumont Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, H. Muckenhirn, D. R. Padfield, J. Qin, D. Rozenberg, T. N. Sainath, J. Schalkwyk, M. Sharifi, M. D. Tadmor, Ramanovich, M. Tagliasacchi, A. Tudor, M. Velimirov… find our card
  140. B. Savoldi, M. Gaido, L. Bentivogli, M. Negri, and M. Turchi. Gender Bias in Machine Translation. Transactions of the Association for Computational Linguistics, 9:845–874, Aug. 2021. ISSN 2307-387X. doi: 10.1162/tacl\_a\_00401. URL https://doi.org/10.1162/tacl\_a\_00401. ↗ our card
  141. B. Schuller, S. Steidl, A. Batliner, F. Burkhardt, L. Devillers, C. MüLler, and S. Narayanan. Paralinguistics in speech and language—state-of-the-art and the challenge. Computer Speech & Language, 27(1):4–39, 2013. find
  142. Seamless Communication, L. Barrault, Y.-A. Chung, M. C. Meglioli, D. Dale, N. Dong, P.-A. Duquenne, H. Elsahar, H. Gong, K. Heffernan, J. Hoffman, C. Klaiber, P. Li, D. Licht, J. Maillard, A. Rakotoarison, K. R. Sadagopan, G. Wenzek, E. Ye, B. Akula, P.-J. Chen, N. E. Hachem, B. Ellis, G. M. Gonzale… find our card
  143. J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan, R. A. Saurous, Y. Agiomvrgiannakis, and Y. Wu. Natural TTS synthesis by conditioning WaveNet on Mel spectrogram predictions. In Proc. ICASSP, pages 4779–4783, 2018. find
  144. J. Shen, Y. Jia, M. Chrzanowski, Y. Zhang, I. Elias, H. Zen, and Y. Wu. Non-attentive tacotron: Robust and controllable neural tts synthesis including unsupervised duration modeling, 2020. URL https://arxiv.org/abs/2010.04301. — [arXiv:2010.04301](https://arxiv.org/abs/2010.04301) ↗
  145. K. Shen, Z. Ju, X. Tan, Y. Liu, Y. Leng, L. He, T. Qin, S. Zhao, and J. Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. ArXiv, abs/2304.09116, 2023. URL https: //api.semanticscholar.org/CorpusID:258187322. — [arXiv:2304.09116](https://arxiv.org/abs/2304.09116) find
  146. K. J. Shih, R. Valle, R. Badlani, A. Lancucki, W. Ping, and B. Catanzaro. RAD-TTS: Parallel flow-based tts with robust alignment learning and diverse synthesis. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021. find
  147. Silero. Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier. https://github.com/snakers4/silero-vad, 2021. ↗
  148. R. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor, R. Weiss, R. Clark, and R. A. Saurous. Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volum… find
  149. D. Snyder, G. Chen, and D. Povey. Musan: A music, speech, and noise corpus. arXiv preprint arXiv:1510.08484, 2015. — [arXiv:1510.08484](https://arxiv.org/abs/1510.08484) find
  150. L. Specia, F. Blain, M. Fomicheva, C. Zerva, Z. Li, V. Chaudhary, and A. F. T. Martins. Findings of the WMT 2021 shared task on quality estimation. In Proceedings of the Sixth Conference on Machine Translation, pages 684–725, Online, Nov. 2021. Association for Computational Linguistics. URL https://… find our card
  151. G. Stanovsky, N. A. Smith, and L. Zettlemoyer. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1164.… find our card
  152. I. G. Sánchez and M. F. Orellana. The construction of moral and social identity in immigrant children’s narratives-in-translation. Linguistics and education, 17(3):209–239, 2006. find
  153. J. Thewlis, S. Haco, and L. Hanu. How ai is learning to identify toxic online content, 2021. URL https://www. scientificamerican.com/article/can-ai-identify-toxic-online-content/. find
  154. A. Tjandra, S. Sakti, and S. Nakamura. Speech-to-speech translation between untranscribed unknown languages. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 593–600. find
  155. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… find
  156. C. Wang, J. Pino, A. Wu, and J. Gu. CoVoST: A diverse multilingual speech-to-text translation corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4197–4203, Marseille, France, May 2020a. European Language Resources Association. ISBN 979-10-95546-34-4. URL https:… find
  157. C. Wang, Y. Wu, L. Lu, S. Liu, J. Li, G. Ye, and M. Zhou. Low Latency End-to-End Streaming Speech Recognition with a Scout Network. In Interspeech 2020, pages 2112–2116. find
  158. ISCA, Oct. 2020b. doi: 10.21437/Interspeech.2020-1292. URL http://www.isca-speech.org/archive/Interspeech\_2020/abstracts/1292.html. — [doi:10.21437/interspeech.2020-1292](https://doi.org/10.21437/interspeech.2020-1292) ↗
  159. C. Wang, A. Wu, J. Gu, and J. Pino. CoVoST 2 and Massively Multilingual Speech Translation. In Proc. Interspeech 2021, pages 2247–2251, 2021. doi: 10.21437/Interspeech.2021-2027. — [doi:10.21437/interspeech.2021-2027](https://doi.org/10.21437/interspeech.2021-2027) find
  160. C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei. Neural codec language models are zero-shot text to speech synthesizers. CoRR, abs/2301.02111, 2023a. URL https://doi.org/10.48550/arXiv.2301.02111. ↗ our card
  161. S. Wang, N. Cooper, M. Eby, and E. S. Jo. From human-centered to social-centered artificial intelligence: Assessing chatgpt’s impact through disruptive events. arXiv preprint arXiv:2306.00227, 2023b. — [arXiv:2306.00227](https://arxiv.org/abs/2306.00227) find
  162. Y. Wang, J. Bai, R. Huang, R. Li, Z. Hong, and Z. Zhao. Speech-to-speech translation with discrete-unit-based style transfer. CoRR, abs/2309.07566, 2023c. find
  163. N. G. Ward, J. E. Avila, E. Rivas, and D. Marco. Dialogs re-enacted across languages, 2023. find
  164. A. Waytz, J. Heafner, and N. Epley. The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle. Journal of experimental social psychology, 52:113–117, 2014. find
  165. Y. Wen, J. Kirchenbauer, J. Geiping, and T. Goldstein. Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust. arXiv preprint arXiv:2305.20030, 2023. — [arXiv:2305.20030](https://arxiv.org/abs/2305.20030) find
  166. J. Zaı̈di, H. Seuté, B. van Niekerk, and M.-A. Carbonneau. Daft-exprt: Cross-speaker prosody transfer on any text for expressive speech synthesis. In Interspeech 2022. ISCA, sep 2022. doi: 10.21437/interspeech.2022-10761. URL https://doi.org/10.21437%2Finterspeech.2022-10761. — [doi:10.21437/interspeech.2022-10761](https://doi.org/10.21437/interspeech.2022-10761) find
  167. N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2022. URL https: //api.semanticscholar.org/CorpusID:236149944. find our card
  168. Y. Zhang, W. Han, J. Qin, Y. Wang, A. Bapna, Z. Chen, N. Chen, B. Li, V. Axelrod, G. Wang, Z. Meng, K. Hu, A. Rosenberg, R. Prabhavalkar, D. S. Park, P. Haghani, J. Riesa, G. Perng, H. Soltau, T. Strohman, B. Ramabhadran, T. Sainath, P. Moreno, C.-C. Chiu, J. Schalkwyk, F. Beaufays, and Y. Wu. Googl… find
  169. Z. Zhang, L. Zhou, J. Ao, S. Liu, L. Dai, J. Li, and F. Wei. SpeechUT: Bridging speech and text with hidden-unit for encoder-decoder based speech-text pre-training. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1663–1676, Abu Dhabi, United Arab Emir… find
  170. Z. Zhang, L. Zhou, C. Wang, S. Chen, Y. Wu, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. CoRR, abs/2303.03926, 2023b. URL https://doi.org/10.48550/arXiv.2303.03926. — [doi:10.48550/arxiv.2303.03926](https://doi.org/10.48550/arxiv.2303.03926) ↗
  171. A. Model Card - Sonar Speech Encoders Model Details a • Person or organization developing model: Developed by FAIR, Meta • Model date: Novemeber 30, 2023 • Model version: 1.0 • Model type: xx – Information about training algorithms, parameters, fairness constraints or other applied approaches, and f… find