Elamur Research⌕
☰
home / papers / 2024 / GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities

GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities

Paper · 2024 · Sreyan Ghosh, Sonal Kumar, Ashish Seth et al.

Summary

Large audio-language models (LALMs) such as LTU, SALMONN and Pengi handle captioning and simple factual audio questions well. However, they connect the audio encoder to the language model through simple linking layers, and they fail on questions that require multi-step inference about non-speech sounds. GAMA addresses both problems. Its LLM receives three kinds of audio evidence as a prefix:

  • output from a pre-trained Audio Q-Former, which carries semantically generalized information;
  • features from an AudioSet-tuned Audio Spectrogram Transformer (AST), merged across several layers by a multi-layer aggregator;
  • during instruction tuning, a soft prompt listing event tags predicted by AST.

The model is first fine-tuned on a large mix of open audio-language data. It is then instruction-tuned on CompA-R, a GPT-generated set of reasoning-heavy instruction-response pairs built from audio metadata and human-written examples. The authors also introduce CompA-R-test, a human-labeled benchmark for open-ended reasoning questions. GAMA outperforms earlier LALMs on classification, captioning, entailment and hallucination tests, giving the field both a stronger model and tools for measuring audio reasoning.

Key points

  • Architecture: an LLM fine-tuned with LoRA. It is fed Audio Q-Former features and multi-layer AST features through MLP projections. The instruction-tuned variant (GAMA-IT) adds a soft prompt containing AST-predicted audio tags.
  • Across 16 datasets covering 4 task types (e.g., ESC-50, DCASE2017, VGGSound, FSD50K, AudioSet, NSynth, GTZAN), GAMA beats prior LALMs by roughly 1%–84%. Removing the AST and aggregator causes the largest drop on these tasks.
  • GAMA achieves the best overall F1 on the audio entailment (deductive reasoning) benchmark, where SALMONN comes closest. It also leads on POPE-style object hallucination evaluation.
  • GAMA-IT gains 4%–50% on OpenAQA, 8%–58% on CompA-R-test and 8%–30% on dense captioning. On these open-ended tasks, the Audio Q-Former is the most important component.
  • Stated limitations: music understanding is evaluated only lightly, larger 13B-scale LLMs were not tried because of compute limits, and the audio encoders are bigger than those in the baselines.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations🏢Adobe SystemsSan Jose · United States35 works · 3,711 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksSpeech & spoken language · 480 works

Cite

Cite as Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami and Dinesh Manocha. 2024. GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.361

@inproceedings{ghosh2024gama,
  title = {{GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities}},
  author = {Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami and Dinesh Manocha},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.361},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.361}
}

References

64 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023a. Clap learning 2018. A dump of BBC’s sound effects library. audio concepts from natural language supervision. This dump was created using the script found at In ICASSP 2023-2023 IEEE International Confer-https://github.com… find
  2. IEEE. 2023. SoundBible - Free Sound Clips, Sound Bites, and Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Sound Effects. Accessed: 25 September 2023. find
  3. Ismail, and Huaming Wang. 2023b. Clap learning audio concepts from natural language supervision. find
  4. Andrea Agostinelli, Timo I. Denk, Zalán Borsos, In ICASSP 2023 - 2023 IEEE International Confer-Jesse Engel, Mauro Verzetti, Antoine Caillon, ence on Acoustics, Speech and Signal Processing Qingqing Huang, Aren Jansen, Adam Roberts, Marco (ICASSP), pages 1–5. find
  5. Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. 2023. Musiclm: Generating music from Jesse Engel, Cinjon Resnick, Adam Roberts, Sander text. find
  6. Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. 2017. Neural audio synthesis of Sébastien Bubeck, Varun Chandrasekaran, Ronen El-musical notes with wavenet autoencoders. In Inter-dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, national Conference on Machine Learning, pages Peter Lee, Yin…
  7. Font, and Xavier Serra. 2021. Fsd50k: an open Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew dataset of human-labeled sound events. IEEE/ACM Zisserman. 2020. Vggsound: A large-scale audio-Transactions on Audio, Speech, and Language Pro-visual dataset. cessing, 30:829–852. find
  8. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919.
  9. Wanyun Cui and Qianle Wang. 2024. Ada-instruct: Adapting instruction generators for complex reasoning. find
  10. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee. find
  11. Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. 2023. Pengi: An audio language model for audio tasks. find
  12. Soham Deshmukh, Benjamin Elizalde, and Huaming Wang. 2022. Audio retrieval with wavtext5k and clap training. arXiv preprint arXiv:2209.14275. — [arXiv:2209.14275](https://arxiv.org/abs/2209.14275) find
  13. Soham Deshmukh, Shuo Han, Hazim Bukhari, Benjamin Elizalde, Hannes Gamper, Rita Singh, and Bhiksha Raj. 2024. Audio entailment: Assessing deductive reasoning for audio understanding. arXiv preprint arXiv:2407.18062. — [arXiv:2407.18062](https://arxiv.org/abs/2407.18062) find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  15. Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. 2020. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736–740. IEEE. find
  16. Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. 2022. Fsd50k: An open dataset of human-labeled sound events. find
  17. Joshua P Gardner, Simon Durand, Daniel Stoller, and Rachel M Bittner. 2024. LLark: A multimodal foundation model for music. find
  18. Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 77… find
  19. Sreyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru, Ramani Duraiswami, and Dinesh Manocha. 2024a. Recap: Retrieval-augmented audio captioning. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1161–1165. find
  20. Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Reddy Evuru, Ramaneswaran S, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024b. Compa: Addressing the gap in compositional reasoning in audio-language models. In The Twelfth International Conference on Learning Re… find
  21. Yuan Gong, Yu-An Chung, and James Glass. 2021. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778. find
  22. Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James R. Glass. 2024. Listen, think, and understand. In The Twelfth International Conference on Learning Representations. find
  23. Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R. Glass. 2023. Contrastive audio-visual masked autoencoder. In The Eleventh International Conference on Learning Representations. find
  24. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.
  25. Yuan Gong, Jin Yu, and James Glass. 2022. Vocalsound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 151–155. IEEE. find
  26. Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. 2022. Clothoaqa: A crowdsourced dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pages 1140–1144. IEEE. find
  27. Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305.15717.
  28. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744.
  29. Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2022. Audioclip: Extending clip to image, text and audio. In ICASSP 2022. find
  30. Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2021. Audioclip: Extending clip to image, text and audio. find
  31. Yuchen Hu, CHEN CHEN, Chao-Han Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen, and Ensiong Chng. 2024. Large language models are efficient learners of noise-robust speech recognition. In The Twelfth International Conference on Learning Representations. find
  32. Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. 2024. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38… find
  33. Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pa…
  34. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  35. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning.
  36. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36.
  37. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. 2023c. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499.
  38. Vincent Lostanlen, Carmine-Emanuele Cella, Rachel Bittner, and Slim Essid. 2018. Medley-solos-db: a crosscollection dataset for musical instrument recognition. Zenodo. find
  39. Vincent Lostanlen, Carmine-Emanuele Cella, Rachel Bittner, and Slim Essid. 2019. Medley-solos-DB: a cross-collection dataset for musical instrument recognition. find
  40. Annamaria Mesaros, Toni Heittola, Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Emmanuel Vincent, Bhiksha Raj, and Tuomas Virtanen. 2017. find
  41. Dcase 2017 challenge setup: Tasks, datasets and baseline system. In DCASE 2017-Workshop on Detection and Classification of Acoustic Scenes and Events. find
  42. Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen. 2018. A multi-device dataset for urban acoustic scene classification. arXiv preprint arXiv:1807.09840. — [arXiv:1807.09840](https://arxiv.org/abs/1807.09840) find
  43. Irene Martin Morato and Annamaria Mesaros. 2021. Macs - multi-annotator captioned soundscapes. find
  44. Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. 2024. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. find
  45. Junwoo Park, Youngwoo Cho, Gyuhyeon Sim, Hojoon Lee, and Jaegul Choo. 2022. Enemy spotted: in-game gun sound dataset for gunshot classification and localization. In 2022 IEEE Conference on Games (CoG), pages 56–63. IEEE. find
  46. Chun-Yi Kuan, Wei-Ping Huang, and Hung-yi Lee. 2024. Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models. arXiv preprint arXiv:2406.08402. — [arXiv:2406.08402](https://arxiv.org/abs/2406.08402) find
  47. Karol J Piczak. 2015. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015– 1018. find
  48. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PM…
  49. Yaman Kumar Singla, Jui Shah, Changyou Chen, and Rajiv Ratn Shah. 2022. What do audio transformers hear? probing their representations for language delivery & structure. In 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pages 910–925. IEEE. find
  50. Sonniss Limited. 2022. Sonniss Game Audio. Registered in England, UK. Company number: 09377364. Accessed: 25 September 2023. find
  51. Luoyi Sun, Xuenan Xu, Mengyue Wu, and Weidi Xie. 2023. A large-scale dataset for audio-language representation learning. arXiv preprint arXiv:2309.11500. — [arXiv:2309.11500](https://arxiv.org/abs/2309.11500) find
  52. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. 2024. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations.
  53. Mi Tian, Ajay Srinivasamurthy, Mark Sandler, and Xavier Serra. 2014. A study of instrument-wise onset detection in beijing opera percussion ensembles. In 2014 ieee international conference on acoustics, speech and signal processing (icassp), pages 2159– 2163. IEEE. find
  54. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  55. George Tzanetakis, Georg Essl, and Perry Cook. 2001. Automatic musical genre classification of audio signals. find
  56. Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. 2018. Espnet: End-to-end speech processing toolkit. arXiv preprint arXiv:1804.00015. — [arXiv:1804.00015](https://arxiv.org/abs/1804.00015) find
  57. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
  58. Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023a. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP. find
  59. Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023b. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP. find
  60. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations.
  61. Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. 2024. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601.
  62. Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792.
  63. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.
  64. Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ghosh-2024-gama-large-audio-language-model/