Elamur Research
☰
home / papers / 2023 / SALMONN: Towards Generic Hearing Abilities for Large Language Models

SALMONN: Towards Generic Hearing Abilities for Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2310.13289

Summary

Most multimodal LLMs that accept sound handle only one kind of input: speech, environmental audio events, or music. That leaves them short of general machine hearing. SALMONN covers all three in a single end-to-end model. It pairs two frozen encoders, Whisper for speech and BEATs for non-speech audio, and concatenates their 50Hz features frame by frame. A window-level Q-Former converts these features into a variable number of time-ordered audio tokens. Those tokens go, together with a text instruction, into a frozen Vicuna LLM fitted with LoRA adaptors. Training involves cross-modal pre-training and instruction tuning on speech, audio and music tasks. The authors found that this locks the model into trained tasks such as ASR and captioning, a problem they call task over-fitting. To fix it, they add a cheap few-shot activation tuning stage that restores the LLM's emergent abilities. On a three-level benchmark suite, SALMONN handles both trained and unseen tasks competitively, which suggests generic hearing for LLMs is feasible. Code, checkpoints and data are released.

Key points

  • Architecture:
    • Whisper speech encoder and BEATs audio encoder, both frozen and running at 50Hz.
    • A window-level Q-Former maps each window of L frames to N tokens. This keeps audio-text alignment monotonic and fine-grained in time.
    • The Vicuna LLM stays frozen; only the Q-Former and LoRA are trained.
  • Training: three stages:
    • cross-modal pre-training
    • instruction tuning
    • few-shot activation tuning
  • Task over-fitting: the authors study this problem by varying the LoRA scaling factor. Activation tuning also reduces catastrophic forgetting of trained tasks.
  • Evaluation levels:
    • Level 1: eight trained tasks, such as ASR, speech translation, audio captioning, emotion recognition and speaker verification.
    • Level 2: five untrained speech-based NLP tasks, such as translation into unseen languages, slot filling and spoken-query QA.
    • Level 3: tasks needing non-speech understanding. These include two newly proposed ones, audio-based storytelling and speech audio co-reasoning (SAC).
  • Resources: GPT-3.5 was used to generate audio/speech QA data and to judge SQQA and SAC outputs. The release is at github.com/bytedance/SALMONN (Tsinghua University & ByteDance).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

182 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 179 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.13289 · arXiv 2310.13289

@misc{tang2023salmonn,
  title = {{SALMONN: Towards Generic Hearing Abilities for Large Language Models}},
  author = {Tang, Changli and Yu, Wenyi and Sun, Guangzhi and Chen, Xianzhao and Tan, Tian and Li, Wei and Lu, Lu and Ma, Zejun and Zhang, Chao},
  year = {2023},
  doi = {10.48550/arxiv.2310.13289},
  eprint = {2310.13289},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.13289}
}

References

62 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. MusicLM: Generating music from text. arXiv:2301.11325, 2023. — [arXiv:2301.11325](https://arxiv.org/abs/2301.11325) find
  2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, et al. Flamingo: a visual language model for few-shot learning. In Proc. NeurIPS, New Orleans, 2022. find our card
  3. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. PaLM 2 technical report. arXiv:2305.10403, 2023. find our card
  4. Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: A spoken language understanding resource package. In Proc. EMNLP, 2020. find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proc. NeurIPS, New Orleans, 2020. find
  6. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335–359, 2008. find
  7. Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-LLM: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv:2305.04160, 2023a. — [arXiv:2305.04160](https://arxiv.org/abs/2305.04160) find
  8. Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Yu Qiao, Tong Lu, et al. VideoLLM: Modeling video sequence with large language models. arXiv:2305.13292, 2023b. — [arXiv:2305.13292](https://arxiv.org/abs/2305.13292) find
  9. Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. GigaSpeech: An evolving, multi-domain ASR corpus with 10,000 hours of transcribed audio. In Proc. Interspeech, Brno, 2021. find
  10. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. WavLM: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505– 1518, 2022. find
  11. Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In Proc. ICML, Honolulu, 2023c. find
  12. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, March 2023. URL https: //lmsys.org/blog/2023-03-30-vicuna… find
  13. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, et al. Scaling instruction-finetuned language models. arXiv:2210.11416, 2022. find
  14. Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent. LibriMix: An open-source dataset for generalizable speech separation. arXiv:2005.11262, 2020. — [arXiv:2005.11262](https://arxiv.org/abs/2005.11262) find
  15. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, et al. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. arXiv:2305.06500, 2023. find our card
  16. SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. LP-MusicCaps: LLM-based pseudo music captioning. arXiv:2307.16372, 2023. — [arXiv:2307.16372](https://arxiv.org/abs/2307.16372) find
  17. Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In Proc. ICASSP, Barcelona, 2020. find
  18. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General language model pretraining with autoregressive blank infilling. In Proc. ACL, Dublin, Ireland, 2022. find our card
  19. Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, et al. Prompting large language models with speech recognition abilities. arXiv preprint arXiv:2307.11795, 2023. — [arXiv:2307.11795](https://arxiv.org/abs/2307.11795) find
  20. Yuan Gong, Sameer Khurana, Leonid Karlinsky, and James Glass. Whisper-AT: Noise-robust automatic speech recognizers are also strong general audio event taggers. In Proc. Interspeech, Dublin, Ireland, 2023a. find
  21. Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv:2305.10790, 2023b. — [arXiv:2305.10790](https://arxiv.org/abs/2305.10790) find
  22. Haohan Guo, Shaofei Zhang, Frank K. Soong, Lei He, and Lei Xie. Conversational end-to-end TTS for voice agents. In Proc. SLT, 2021. find
  23. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of large language models. In Proc. ICLR, 2022. find
  24. Chien-yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi-Yuan Hsiao, Chun-Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, et al. Dynamic-SUPERB: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech. arXiv:2309.09510, 2023a. — [arXiv:2309.09510](https://arxiv.org/abs/2309.09510) find
  25. Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. AudioGPT: Understanding and generating speech, music, sound, and talking head. arXiv:2304.12995, 2023b. — [arXiv:2304.12995](https://arxiv.org/abs/2304.12995) find
  26. Zili Huang, Desh Raj, Paola Garcı́a, and Sanjeev Khudanpur. Adapting self-supervised models to multi-talker speech recognition using speaker embeddings. In Proc. ICASSP, Rhodes, Greek, 2023c. find
  27. Anette Hulth. Improved automatic keyword extraction given more linguistic knowledge. In Proc. EMNLP, Sapporo, Japan, 2003. find
  28. Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generating captions for audios in the wild. In Proc. NAACL-HLT, Minneapolis, 2019. find
  29. Harlen Lane and Bernard Tranel. The Lombard sign and the role of hearing in speech. Journal of Speech Hearing Research, 14:677–709, 1971. find
  30. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proc. ICML, Hawaii, 2023a. find our card
  31. Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, et al. MERT: Acoustic music understanding model with large-scale self-supervised training. arXiv:2306.00107, 2023b. — [arXiv:2306.00107](https://arxiv.org/abs/2306.00107) find
  32. Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. Music understanding LLaMA: Advancing text-to-music generation with question answering and captioning. arXiv:2308.11276, 2023. — [arXiv:2308.11276](https://arxiv.org/abs/2308.11276) find
  33. Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, et al. Macaw-LLM: Multi-modal language modeling with image, audio, video, and text integration. arXiv:2306.09093, 2023. find
  34. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. arXiv:2306.05424, 2023. find our card
  35. Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. WavCaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. arXiv:2303.17395, 2023. — [arXiv:2303.17395](https://arxiv.org/abs/2303.17395) find
  36. Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman. Voxceleb: Large-scale speaker verification in the wild. Computer Speech & Language, 60:101027, 2019. find
  37. Chaitanya Narisetty, Emiru Tsunoo, Xuankai Chang, Yosuke Kashiwagi, Michael Hentschel, and Shinji Watanabe. Joint speech recognition and audio captioning. In Proc. ICASSP, Singapore, 2022. find
  38. OpenAI. GPT-4 technical report. arXiv:2303.08774, 2023. find our card
  39. Pilar Oplustil-Gallegos, Johannah O’Mahony, and Simon King. Comparing acoustic and textual representations of previous linguistic context for improving text-to-speech. In Proc. SSW, 2021. find
  40. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Proc. NeurIPS, New Orleans, 2022. find
  41. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In Proc. ICASSP, South Brisbane, 2015. find
  42. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. arXiv:2304.03277, 2023. find our card
  43. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proc. ICML, Honolulu, 2023. find our card
  44. Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. AudioPaLM: A large language model that can speak and listen. arXiv:2306.12925, 2023. find our card
  45. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. PandaGPT: One model to instruction-follow them all. arXiv:2305.16355, 2023. find
  46. Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Fine-grained audio-visual joint representations for multimodal large language models. arXiv preprint arXiv:2310.05863, 2023. — [arXiv:2310.05863](https://arxiv.org/abs/2310.05863) find
  47. John Thickstun, Zaid Harchaoui, and Sham Kakade. Learning features of music from scratch. In Proc. ICLR, Toulon, France, 2017. find
  48. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, et al. LLaMA: Open and efficient foundation language models. arXiv:2302.13971, 2023. find
  49. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. NeurIPS, Long Beach, 2017. find
  50. Changhan Wang, Anne Wu, and Juan Pino. CoVoST 2 and massively multilingual speech translation. In Proc. Interspeech, Brno, Czech Republic, 2021. find
  51. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In Proc. ICLR, 2022a. find
  52. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022b. find our card
  53. Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, et al. On decoder-only architecture for speech-to-text and large language model integration. arXiv:2307.03917, 2023. — [arXiv:2307.03917](https://arxiv.org/abs/2307.03917) find
  54. Wen Wu, Chao Zhang, and Philip C Woodland. Emotion recognition by fusing time synchronous and time asynchronous representations. In Proc. ICASSP, Toronto, Canada, 2021. find
  55. Guanghui Xu, Wei Song, Zhengchen Zhang, Chao ZHang, Xiaodong He, and Bowen Zhou. Improving prosody modelling with cross-utterance BERT embeddings for end-to-end speech synthesis. In Proc. ICASSP, 2021. find
  56. Yi Yang, Wen-tau Yih, and Christopher Meek. WikiQA: A challenge dataset for open-domain question answering. In Proc. EMNLP, Lisbon, Portugal, 2015. find our card
  57. Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Connecting speech encoder and large language model for asr. arXiv preprint arXiv:2309.13963, 2023. — [arXiv:2309.13963](https://arxiv.org/abs/2309.13963) find
  58. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv:2305.11000, 2023a. find our card
  59. Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. arXiv:2306.02858, 2023b. find our card
  60. Ya-Jie Zhang, Chao Zhang, Wei Song, Zhengchen Zhang, Yonghui Wu, and Xiaodong He. Prosody modelling with pre-trained cross-utterance representations for improved speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2812–2823, 2023c. find
  61. Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In Proc. CVPR, New Orleans, 2022. find
  62. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv:2304.10592, 2023. find our card