📄 SLM: Bridge the thin gap between speech and text foundation models

Paper · 2023 · Mingqiu Wang, Wei Han, Izhak Shafran et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

🏛️ Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations🏢Google DeepMindLondon · United Kingdom56 works · 78,880 citations

🧭 Fields, topics and concepts

Cite

Cite as Mingqiu Wang, Wei Han, Izhak Shafran, Zelin Wu, Chung-Cheng Chiu, Yuan Cao, Yongqiang Wang, Nanxin Chen, Yu Zhang, Hagen Soltau, Paul Rubenstein, Lukas Zilka, Dian Yu, Zhong Meng, Golan Pundak, Nikhil Siddhartha, Johan Schalkwyk and Yonghui Wu. 2023. SLM: Bridge the thin gap between speech and text foundation models. arXiv:2310.00230.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.00230 · arXiv 2310.00230

@misc{wang2023slm,
  title = {{SLM: Bridge the thin gap between speech and text foundation models}},
  author = {Mingqiu Wang, Wei Han, Izhak Shafran, Zelin Wu, Chung-Cheng Chiu, Yuan Cao, Yongqiang Wang, Nanxin Chen, Yu Zhang, Hagen Soltau, Paul Rubenstein, Lukas Zilka, Dian Yu, Zhong Meng, Golan Pundak, Nikhil Siddhartha, Johan Schalkwyk and Yonghui Wu},
  year = {2023},
  doi = {10.48550/arxiv.2310.00230},
  eprint = {2310.00230},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.00230}
}

References

40 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [1] OpenAI, “Gpt-4 technical report,” 2023.
2. [2] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al., “Palm 2 technical report,” arXiv, 2023.
3. [3] Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, et al., “Google usm: Scaling automatic speech recognition beyond 100 languages,” arXiv, 2023. find
4. [11] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al., “Audiolm: a language modeling approach to audio generation,” IEEE/ACM Transactions on Audio, Speech, and Language Proc…
5. [12] Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang, “Pengi: An audio language model for audio tasks,” arXiv preprint arXiv:2305.11834, 2023. — [arXiv:2305.11834](https://arxiv.org/abs/2305.11834) find
6. [13] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra, “Imagebind: One embedding space to bind them all,” in Proc. CVPR, 2023.
7. [14] Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass, “Listen, think, and understand,” arXiv, 2023. find
8. [4] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proc. ICML, 2023.
9. [15] Guy Yariv, Itai Gat, Lior Wolf, Yossi Adi, and Idan Schwartz, “Audiotoken: Adaptation of text-conditioned diffusion models for audio-to-image generation,” arXiv, 2023. find
10. [5] Mingqiu Wang, Izhak Shafran, Hagen Soltau, Wei Han, Yuan Cao, Dian Yu, and Laurent El Shafey, “Speech-to-text adapter and speech-to-entity retriever augmented llms for speech understanding,” arXiv, 2023. find
11. [16] Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino, “CoVoST 2 and Massively Multilingual Speech Translation,” in Proc. Interspeech, 2021. find
12. [6] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, “Parameter-efficient transfer learning for nlp.,” in Proc. ICML, 2019. find
13. [7] Hagen Soltau, Izhak Shafran, Mingqiu Wang, Abhinav Rastogi, Jeffrey Zhao, Ye Jia, Wei Han, Yuan Cao, and Aramys Miranda, “Speech aware dialog system technology challenge (dstc11),” Proc. Interspeech, 2023. find
14. [8] Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu, “Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities,” arXiv, 2023.
15. [9] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021.
16. [10] Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al., “Audiopalm: A large language model that can speak and listen,” arXiv, 2023.
17. [17] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto, “Stanford alpaca: An instruction-following llama model,” 2023. find
18. [18] Ye Jia, Heiga Zen, Jonathan Shen, Yu Zhang, and Yonghui Wu, “Png BERT: augmented BERT on phonemes and graphemes for neural TTS,” CoRR, 2021. find
19. [19] Jonathan Shen, Ye Jia, Mike Chrzanowski, Yu Zhang, Isaac Elias, Heiga Zen, and Yonghui Wu, “Non-attentive tacotron: Robust and controllable neural tts synthesis including unsupervised duration modeling,” arXiv, 2020. find
20. [20] Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu, “Self-supervised learning with random-projection quantizer for speech recognition,” in Proc. ICML, 2022. find
21. [21] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, 2020.
22. [22] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al., “Crosslingual generalization through multitask finetuning,” arXiv, 2022.
23. [23] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel, “mt5: A massively multilingual pre-trained text-to-text transformer,” arXiv, 2020.
24. [24] William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoc Le, and Mohammad Norouzi, “Speechstew: Simply mix all available speech recognition data to train one large neural network,” arXiv, 2021. find
25. [25] Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux, “Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” arXiv, 2021.
26. [26] Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” in IEEE Spoken Language Technology Workshop (SLT), 2022.
27. [27] Andrew Cameron Morris, Viktoria Maier, and Phil Green, “From wer and ril to mer and wil: improved evaluation measures for connected speech recognition,” in Proc. International Conference on Spoken Language Processing, 2004. find
28. [28] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002.
29. [29] Matt Post, “A call for clarity in reporting bleu scores,” arXiv, 2018.
30. [30] Tsendsuren Munkhdalai, Zelin Wu, Golan Pundak, Khe Chai Sim, Jiayang Li, Pat Rondon, and Tara N Sainath, “Nam+: Towards scalable end-to-end contextual biasing for adaptive asr,” in 2022 IEEE Spoken Language Technology Workshop (SLT), 2023. find
31. [31] Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau, “mslam: Massively multilingual joint pre-training for speech and text,” arXiv, 2022. find
32. [32] Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro Moreno, Ankur Bapna, and Heiga Zen, “Maestro: Matched speech text representations through modality matching,” arXiv, 2022. find
33. [33] Yong Cheng, Yu Zhang, Melvin Johnson, Wolfgang Macherey, and Ankur Bapna, “Mu2 slam: Multitask, multilingual speech and language models,” in Proc. ICML, 2023. find
34. [34] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al., “Natural questions: a benchmark for question answering research,” Transactions of the Association for Computational Linguist…
35. [35] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, “Lora: Low-rank adaptation of large language models,” arXiv, 2021.
36. [36] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al., “Flamingo: a visual language model for few-shot learning,” Proc. Neurips, 2022.
37. [37] Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit Iyyer, and Noah Constant, “Overcoming catastrophic forgetting in zero-shot cross-lingual generation,” arXiv, 2022. find
38. [38] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al., “Scaling instruction-finetuned language models,” arXiv, 2022. find
39. [39] Yoshua Bengio, Réjean Ducharme, and Pascal Vincent, “A neural probabilistic language model,” Proc. Neurips, 2000. find
40. [40] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv, 2018.

Updated 30 October 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2023-slm-bridge-thin-gap-between/