Elamur Research⌕
☰
home / papers / 2024 / SpeechVerse: A Large-scale Generalizable Audio Language Model

SpeechVerse: A Large-scale Generalizable Audio Language Model

Paper · 2024 · Nilaksh Das, Saket Dingliwal, Srikanth Ronanki et al.

Summary

Most LLM-based audio models handle only a few fine-tuned tasks, such as ASR or translation. Cascaded ASR-then-LLM pipelines also lose paralinguistic cues like tone or emotion. SpeechVerse, from AWS AI Labs, addresses both problems by connecting a frozen self-supervised speech encoder to a frozen text LLM (FlanT5). Only a small set of parameters is trained:

  • learnable weights that blend the encoder's intermediate layers,
  • a 1-D convolution stack that shortens the audio features and maps them into the LLM's embedding space,
  • LoRA adapters.

Training unstable gradients are avoided with a two-stage curriculum. The model first learns audio-text alignment on ASR alone, then warms up LoRA on ASR, and only then adds the remaining tasks. Instruction finetuning covers 11 tasks spanning ASR, spoken language understanding and paralinguistics, each with many prompt wordings.

The single multi-task model outperforms task-specific baselines on 9 of 11 tasks. It also copes with out-of-domain data, new prompts and unseen tasks. This shows that speech abilities can be added to an LLM cheaply without losing its instruction following.

Key points

  • The 11 tasks are ASR, ST, IC, SF, KWE, KWS, ER, ASC, SC, AC and SNS.
  • Training data includes Librispeech (960 h), Mozilla Common Voice 5.1, VoxPopuli, SLURP, EuroParl, MSP-Podcast 1.11 and CoVoST2.
  • The convolution blocks use kernel size 3 and downsample audio to 12.5 Hz, so each frame covers 80 ms.
  • Each task is trained with 30+ instruction variants. Constrained and joint decoding improve results on unseen tasks by up to 21% absolute.
  • The authors note several limitations:
    • only one LLM backbone (FlanT5) was tested;
    • generality on new tasks trades off against specialization on training tasks;
    • performance on unseen tasks was not quantitatively measured.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AmazonSeattle · United States128 works · 41,697 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSpeech & spoken language · 519 works

Cite

Cite as Nilaksh Das, Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, Zhaocheng Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, Xilai Li, Karel Mundnich, Monica Sunkara, Sravan Bodapati, Sundararajan Srinivasan, Kyu J Han and Katrin Kirchhoff. 2024. SpeechVerse: A Large-scale Generalizable Audio Language Model. arXiv:2405.08295.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.08295 · arXiv 2405.08295

@misc{das2024speechverse,
  title = {{SpeechVerse: A Large-scale Generalizable Audio Language Model}},
  author = {Nilaksh Das, Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, Zhaocheng Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, Xilai Li, Karel Mundnich, Monica Sunkara, Sravan Bodapati, Sundararajan Srinivasan, Kyu J Han and Katrin Kirchhoff},
  year = {2024},
  doi = {10.48550/arxiv.2405.08295},
  eprint = {2405.08295},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.08295}
}

References

31 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Brown et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020. A. Chowdhery et al., “Palm: Scaling language modeling with pathways,” Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023. A. Radford, K. Nar…
  2. R. Lotfian and C. Busso, “Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,” IEEE Transactions on Affective Computing, vol. 10, no. 4, pp. 471–483, 2017. find
  3. C. Wang, A. Wu, and J. Pino, “Covost 2 and massively multilingual speech-to-text translation,” arXiv preprint arXiv:2007.10310, 2020.
  4. e. a. Cieri Christopher, “Fisher english training speech part 1 speech ldc2004s13,” Web Download. Philadelphia: Linguistic Data Consortium, 2004. find
  5. e. a. Cieri Christopher, “Fisher english training part 2, speech ldc2005s13,” Web Download. Philadelphia: Linguistic Data Consortium, 2005. find
  6. R. Taori et al., Stanford alpaca: An instruction-following llama model, 2023. find
  7. S. Chen et al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, pp. 1505–1518, 2021. find
  8. C.-C. Chiu, J. Qin, Y. Zhang, J. Yu, and Y. Wu, “Self-supervised learning with random-projection quantizer for speech recognition,” in International Conference on Machine Learning, PMLR, 2022, pp. 3915–3924. find
  9. A. Bapna et al., “Mslam: Massively multilingual joint pre-training for speech and text,” arXiv preprint arXiv:2202.01374, 2022. find
  10. Seamless Communication et al., Seamlessm4t: Massively multilingual & multimodal machine translation, 2023. arXiv: 2308.11596 [cs.CL].
  11. X. Li et al., “Multilingual speech translation from efficient finetuning of pretrained models,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), C. Zong, F. X… find
  12. S. Seo, D. Kwak, and B. Lee, “Integration of pre-trained networks with continuous token interface for end-to-end spoken language understanding,” in Proc. ICASSP, 2022, pp. 7152– 7156. find
  13. Y. Wang, A. Boumadane, and A. Heba, “A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding,” arXiv preprint arXiv:2111.02735, 2022. — [arXiv:2111.02735](https://arxiv.org/abs/2111.02735) find
  14. A. Derington, H. Wierstorf, A. Özkil, F. Eyben, F. Burkhardt, and B. W. Schuller, “Testing speech emotion recognition machine learning models,” arXiv preprint arXiv:2312.06270, 2023. — [arXiv:2312.06270](https://arxiv.org/abs/2312.06270) find
  15. B. T. Willard and R. Louf, “Efficient guided generation for llms,” arXiv preprint arXiv:2307.09702, 2023. find
  16. J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 824–24 837, 2022.
  17. L. Kaiser et al., “One model to learn them all,” arXiv preprint arXiv:1706.05137, 2017. find
  18. C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020.
  19. Y.-C. Chen et al., “Speechnet: A universal modularized model for speech processing tasks,” arXiv preprint arXiv:2105.03070, 2021. — [arXiv:2105.03070](https://arxiv.org/abs/2105.03070) find
  20. J.-B. Alayrac et al., “Flamingo: A visual language model for few-shot learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 23 716–23 736, 2022.
  21. J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International Conference on Machine Learning, PMLR, 2022, pp. 12 888–12 900.
  22. J. Y. Koh, R. Salakhutdinov, and D. Fried, “Grounding language models to images for multimodal inputs and outputs,” in International Conference on Machine Learning, PMLR, 2023, pp. 17 283–17 300. find
  23. Z. Peng et al., “Kosmos-2: Grounding multimodal large language models to the world,” arXiv preprint arXiv:2306.14824, 2023.
  24. K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,” International Journal of Computer Vision, vol. 130, no. 9, pp. 2337–2348, 2022. find
  25. S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,” arXiv preprint arXiv:2305.11834, 2023. — [arXiv:2305.11834](https://arxiv.org/abs/2305.11834) find
  26. Y. Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass, “Listen, think, and understand,” arXiv preprint arXiv:2305.10790, 2023. — [arXiv:2305.10790](https://arxiv.org/abs/2305.10790) find
  27. Y. Shu et al., “Llasm: Large language and speech model,” arXiv preprint arXiv:2308.15930, 2023. — [arXiv:2308.15930](https://arxiv.org/abs/2308.15930) find
  28. J. Iranzo-Sánchez et al., “Europarl-st: A multilingual corpus for speech translation of parliamentary debates,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 8229–8233. find
  29. R. Lotfian and C. Busso, “Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,” IEEE Transactions on Affective Computing, vol. 10, no. 4, pp. 471–483, Oct. 2019. DOI : 10.1109/TAFFC.2017.2736999. — [doi:10.1109/taffc.2017.2736999](https://doi.org/10.1109/taffc.2017.2736999) find
  30. R. Paturi, S. Srinivasan, and X. Li, “Lexical Speaker Error Correction: Leveraging Language Models for Speaker Diarization Error Correction,” in Proc. INTERSPEECH 2023, 2023, pp. 3567–3571. DOI : 10.21437/Interspeech.2023-1982. — [doi:10.21437/interspeech.2023-1982](https://doi.org/10.21437/interspeech.2023-1982) find
  31. e. a. Cieri Christopher, “2000 hub5 english evaluation speech ldc2002s09,” Web Download. Philadelphia: Linguistic Data Consortium, 2002. *Table 11: Various hyper-parameters for our models* Parameter Multitask-WLM Multitask-BRQ Audio encoder LLM Convolution blocks Kernel sizes Strides Audio encoder a… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/das-2024-speechverse-large-scale-generalizable-audio/