Elamur Research⌕
☰
home / papers / 2024 / MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response

MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response

Summary

Music information retrieval usually depends on discriminative models that assign tags from a fixed label set. This limits them for tasks that call for free-form language, such as describing tracks or answering questions about them. MusiLingo addresses this by linking a frozen MERT music encoder to a frozen Vicuna LLM with one trainable projection layer. The layer compresses music embeddings over time and feeds them into the LLM's input layer. This is simpler than the deeper adapter used in the concurrent MU-LLaMA.

Training has two stages. The model is first pre-trained on large-scale captions from LP-MusicCaps-MSD. It is then instruction-tuned on MusicInstruct (MI), a new Q&A resource the authors generated with ChatGPT from MusicCaps captions and then filtered.

On MusicCaps captioning, MusiLingo leads on some metrics but still lags behind the strongest systems on BERT-score. It is competitive on music question answering. The work matters because it adds an openly released instruction dataset and a lightweight recipe for music-text dialogue.

Key points

  • Architecture: MERT (frozen) feeds a single linear projection, which feeds Vicuna (frozen). The design follows the adapter style of MiniGPT-4 and LLaVA and permits larger batch sizes.
  • Pre-training data: LP-MusicCaps-MSD contains about 520k 30-second clips from the ECALS subset of the Million Song Dataset. It uses 1054 labels, averages about 10.2 labels per clip, and provides GPT-3.5 pseudo-captions.
  • MusicInstruct (MI): The dataset has 60,493 Q&A pairs derived from MusicCaps captions. Pairs were produced through few-shot prompting and then filtered by a second ChatGPT check plus removal of malformed outputs.
    • v1 holds 27,540 short, specific Q&A pairs covering tempo, mood, instruments, genre and similar attributes.
    • v2 holds general, longer Q&A pairs.
    • The dataset is released on Hugging Face under cc-by-nc-4.0.
  • Results and ablation:
    • On MusicCaps captioning, MusiLingo is best on B-U and R-L but behind SOTA on BERT-score.
    • Fine-tuning on a mix of short and long MI questions did worse than fine-tuning on a single partition.
    • Short questions help zero-shot MusicQA, while long questions help captioning.
  • Limitations: The model performs well mainly on the dataset it was tuned on, rather than generalizing across Q&A datasets. Subjective long answers generated from captions alone may contain hallucinations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

12 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-naacl.231 · arXiv 2309.08730

@inproceedings{deng2024musilingo,
  title = {{MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response}},
  author = {Deng, Zihao and Ma, Yinghao and Liu, Yudong and Guo, Rongchen and Zhang, Ge and Chen, Wenhu and Huang, Wenhao and Benetos, Emmanouil},
  booktitle = {Findings of the Association for Computational Linguistics: NAACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.231},
  eprint = {2309.08730},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.findings-naacl.231}
}

References

37 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. 2023. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325. — [arXiv:2301.11325](https://arxiv.org/abs/2301.11325) find
  2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock… find our card
  3. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72. find our card
  4. Thierry Bertin-Mahieux, Daniel PW Ellis, Brian Whitman, and Paul Lamere. 2011. The million song dataset. find
  5. ISMIR 2011: Proceedings of the 12th International Society for Music Information Retrieval Conference, October 24-28, 2011, Miami, Florida. find
  6. Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. 2019. The mtg-jamendo dataset for automatic music tagging. In Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML). find
  7. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. find
  8. Tian Cai, Michael I Mandel, and Di He. 2020. Music autotagging as captioning. In Proceedings of the 1st https://www.baskerville.ac.uk/ ↗
  9. Workshop on NLP for Music and Audio (NLP4MusA), pages 67–72. Association for Computational Linguistics. find
  10. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  11. Keunwoo Choi, George Fazekas, Brian McFee, Kyunghyun Cho, and Mark Sandler. 2016. Towards music captioning: Generating music playlist descriptions. arXiv preprint arXiv:1608.04868. — [arXiv:1608.04868](https://arxiv.org/abs/1608.04868) find
  12. SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. 2023a. Lp-musiccaps: Llm-based pseudo music captioning. ISMIR. find
  13. Seungheon Doh, Junwon Lee, and Juhan Nam. 2021. Music playlist title generation: A machine-translation approach. In Proceedings of the 2nd Workshop on NLP for Music and Spoken Audio (NLP4MusA), pages 27–31. find
  14. SeungHeon Doh, Minz Won, Keunwoo Choi, and Juhan Nam. 2023b. Toward universal text-to-music retrieval. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  15. SeungHeon Doh, Minz Won, Keunwoo Choi, and Juhan Nam. 2023c. Toward universal text-to-music retrieval. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  16. Giovanni Gabbolini, Romain Hennequin, and Elena Epure. 2022. Data-efficient playlist captioning with musical and linguistic knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11401–11415. Association for Computational Linguistics. find
  17. Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. 2023a. Joint audio and speech understanding. IEEE Automatic Speech Recognition and Understanding Workshop. find
  18. Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. 2023b. Listen, think, and understand. arXiv preprint arXiv:2305.10790. — [arXiv:2305.10790](https://arxiv.org/abs/2305.10790) find
  19. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations. find
  20. Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel PW Ellis. 2022. Mulan: A joint embedding of music audio and natural language. arXiv preprint arXiv:2208.12415. — [arXiv:2208.12415](https://arxiv.org/abs/2208.12415) find
  21. Haven Kim, Seungheon Doh, Junwon Lee, and Juhan Nam. 2023. Music playlist title generation using artist information. In The AAAI-23 Workshop on Creative AI Across Modalities. find
  22. Edith Law, Kris West, Michael I Mandel, Mert Bay, and J Stephen Downie. 2009. Evaluation of algorithms using games: The case of music tagging. In ISMIR, pages 387–392. Citeseer. find
  23. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597. find our card
  24. Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, et al. 2023b. Mert: Acoustic music understanding model with large-scale self-supervised training. arXiv preprint arXiv:2306.00107. — [arXiv:2306.00107](https://arxiv.org/abs/2306.00107) find
  25. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  26. Chin-Yew Lin and Franz Josef Och. 2004. ORANGE: a method for evaluating automatic evaluation metrics for machine translation. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 501–507, Geneva, Switzerland. COLING. find our card
  27. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning. find our card
  28. Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. 2023b. Music understanding llama: Advancing text-to-music generation with question answering and captioning. arXiv preprint arXiv:2308.11276. — [arXiv:2308.11276](https://arxiv.org/abs/2308.11276) find
  29. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2023. Video-chatgpt: Towards detailed video understanding via large vision and language models. find our card
  30. Ilaria Manco, Emmanouil Benetos, Elio Quinton, and George Fazekas. 2022. Contrastive audio-language learning for music. In Ismir 2022 Hybrid Conference. find
  31. Ilaria Manco, Emmanouil Benetos, Elio Quinton, and György Fazekas. 2021. Muscaps: Generating captions for music audio. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE. find
  32. Xinhao Mei, Xubo Liu, Qiushi Huang, Mark D Plumb-ley, and Wenwu Wang. 2021. Audio captioning transformer. arXiv preprint arXiv:2107.09817. — [arXiv:2107.09817](https://arxiv.org/abs/2107.09817) find
  33. OpenAI. 2023. Gpt-4 technical report. find our card
  34. Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. find our card
  35. Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. 2023. Unified model for image, video, audio and language tasks. arXiv preprint arXiv:2307.16184. — [arXiv:2307.16184](https://arxiv.org/abs/2307.16184) find
  36. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. find
  37. Minz Won, Keunwoo Choi, and Xavier Serra. 2021. Semi-supervised music tagging transformer. International Society for Music Information Retrieval (IS-MIR). find

↑ back to top