Elamur Research⌕
☰
home / papers / 2026 / Thesis Proposal: Multimodal Benchmark for Music Understanding in Large Language Models

Thesis Proposal: Multimodal Benchmark for Music Understanding in Large Language Models

Paper · 2026 · Tomáš Sourada

Summary

This thesis proposal argues that the field lacks a reliable way to test whether multimodal LLMs actually understand music. The author reviews current music benchmarks and finds several problems. They are scattered and mostly test a single modality, usually audio. They lean toward Western repertoire, and their questions can often be answered without perceiving the music itself. Many do not document how prompts were chosen or how answers were extracted, and some rely on proprietary LLMs, which hurts reproducibility and raises data-leakage concerns.

The proposed remedy is an open benchmark of multiple-choice questions spanning four music modalities: audio, symbolic encodings, score images, and lyrics. Hard distractors, sometimes in a different modality, will force models to engage with the content. Prompt selection and answer extraction will be documented, and the pipeline will contain no proprietary parts. Non-Western material will also be included.

Because this is a proposal, no experimental results are reported yet. The planned work follows three research questions:

  1. How to build robust, reproducible evaluation.
  2. How current MLLMs perform across modalities.
  3. Whether the scores models achieve correspond to people's musical skill, validated through comparison with musicians.

Key points

  • The proposal defines music understanding as three combined skills: perception (pitch, timbre, rhythm), knowledge (theory, history, style) and reasoning (harmony, key, meter, form).
  • The symbolic modality covers MIDI, MusicXML and ABC. The author knows of no open-source benchmark that spans more than two musical modalities.
  • Table 1 surveys 14 music-capable models by modality and openness, including M2UGen, ChatMusician, Qwen3-omni, Music Flamingo, GPT-4o, Gemini2.5 and Claude 4.
  • Acknowledged limitations:
    • the closed-QA format itself;
    • scarce open data for non-Western repertoires, with early releases expected to be mostly Western;
    • difficulty generating good distractors;
    • recruiting a diverse pool of musicians.
  • The human study will use informed consent, compensation and IRB approval. The reference list includes the Ukrainian Polyphony Project (Both, Vertiuk, Rybak, 2025).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Charles UniversityPrague · Czechia133 works · 20,686 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Tomáš Sourada. 2026. Thesis Proposal: Multimodal Benchmark for Music Understanding in Large Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-srw.28

@inproceedings{sourada2026thesis,
  title = {{Thesis Proposal: Multimodal Benchmark for Music Understanding in Large Language Models}},
  author = {Tomáš Sourada},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.28},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.28}
}

References

57 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas,… find
  2. Anthropic. 2025. Introducing Claude 4 \\ Anthropic. find
  3. Stefan Balke, Axel Berndt, and Meinard Müller. 2025. ChoraleBricks: A Modular Multitrack Dataset for Wind Music Research. Transactions of the International Society for Music Information Retrieval (TISMIR). find
  4. Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondrej Dusek. 2024. find
  5. Leak, Cheat, Repeat: Brandon James Carone, Iran R. Roman, and Pablo Ripollés. 2025. Evaluating Multimodal Large Language Models on Core Music Perception Tasks. arXiv preprint. ArXiv:2510.22455 [cs]. — [arXiv:2510.22455](https://arxiv.org/abs/2510.22455) find
  6. Jian Chen, Wenye Ma, Penghang Liu, Wei Wang, Tengwei Song, Ming Li, Chenguang Wang, Jiayu Qin, Ruiyi Zhang, and Changyou Chen. 2025. MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models. arXiv preprint. ArXiv:2506.23009 [cs]. — [arXiv:2506.23009](https://arxiv.org/abs/2506.23009) find
  7. Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. 2024. Are We on the Right Way for Evaluating Large Vision-Language Models? arXiv preprint. ArXiv:2403.20330 [cs].
  8. Daniel Chin and Gus Xia. 2025. Language Model Mapping in Multimodal Music Learning: A Grand Challenge Proposal. ArXiv:2503.00427 [cs]. — [arXiv:2503.00427](https://arxiv.org/abs/2503.00427) find
  9. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. Qwen2-Audio Technical Report. arXiv preprint. ArXiv:2407.10759 [eess].
  10. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 other…
  11. Congren Dai, Yue Yang, Krinos Li, Huichi Zhou, Shijie Liang, Zhang Bo, Enyang Liu, Ge Jin, Hongran An, Haosen Zhang, Peiyuan Jing, KinHei Lee, Zhenxuan Zhang, Xiaobing Li, and Maosong Sun. 2025. Musical Score Understanding Benchmark: Evaluating Large Language Models’ Comprehension of Complete Musica… find
  12. Zihao Deng, Yinghao Ma, Yudong Liu, Rongchen Guo, Ge Zhang, Wenhu Chen, Wenhao Huang, and Emmanouil Benetos. 2024. MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response. arXiv preprint. ArXiv:2309.08730 [eess].
  13. Drew Edwards, Simon Dixon, and Emmanouil Benetos. 2023. PiJAMA: Piano Jazz with Automatic MIDI Annotations. Transactions of the International Society for Music Information Retrieval, 6(1). find
  14. Nicola Fanelli, Gennaro Vessio, and Giovanna Castellano. 2025. ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval. arXiv preprint. ArXiv:2507.21917 [cs]. — [arXiv:2507.21917](https://arxiv.org/abs/2507.21917) find
  15. Francesco Foscarin, Andrew McLeod, Philippe Rigaux, Florent Jacquemard, and Masahiko Sakai. 2020. ASAP: A dataset of aligned scores and performances for piano transcription. International Society for Music Information Retrieval Conference (ISMIR 2020), pages 534–541. find
  16. Josh Gardner, Simon Durand, Daniel Stoller, and Rachel M. Bittner. 2024. LLark: A Multimodal Instruction-Following Language Model for Music. arXiv preprint. ArXiv:2310.07160 [cs]. — [arXiv:2310.07160](https://arxiv.org/abs/2310.07160) find
  17. Sreyan Ghosh, Arushi Goel, Lasha Koroshinadze, Sang-gil Lee, Zhifeng Kong, Joao Felipe Santos, Ramani Duraiswami, Dinesh Manocha, Wei Ping, Mohammad Shoeybi, and Bryan Catanzaro. 2025. Music Flamingo: Scaling Music Understanding in Audio Language Models. arXiv preprint. ArXiv:2511.10289 [eess]. — [arXiv:2511.10289](https://arxiv.org/abs/2511.10289) find
  18. Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. 2025. Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models. arXiv preprint. ArXiv:2507.081… — [arXiv:2507.08128](https://arxiv.org/abs/2507.08128) find
  19. Mark Gotham and Peter Jonas. 2025. Score/Lieder: OpenScore lieder corpus v3. find
  20. Open-Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. 2018. Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset. International Conference on Learning Representations. find
  21. Dóra Horváth. 2025. Curtain call for AI: Transforming theatre through technology. Sustainable Futures, 9:100747. find
  22. Jinsheng Huang, Liang Chen, Taian Guo, Fu Zeng, Yusheng Zhao, Bohan Wu, Ye Yuan, Haozhe Zhao, Zhihui Guo, Yichi Zhang, Jingyang Yuan, Wei Ju, Luchen Liu, Tianyu Liu, Baobao Chang, and Ming Zhang. 2025. MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation. arXiv p… find
  23. Anastasiia Ivanova, Natalia Fedorova, Sergei Tilga, and Ekaterina Artemova. 2025. Voices of Freelance Professional Writers on AI: Limitations, Expectations, and Fears. arXiv preprint. ArXiv:2504.05008 [cs]. — [arXiv:2504.05008](https://arxiv.org/abs/2504.05008) find
  24. Jongmin Jung, Dongmin Kim, Sihun Lee, Seola Cho, Hyungjoon Soh, Irmak Bukey, Chris Donahue, and Dasaem Jeong. 2025. Unified Cross-modal Translation of Score Images, Symbolic Music, and Performance Audio. arXiv preprint. ArXiv:2505.12863 [cs]. — [arXiv:2505.12863](https://arxiv.org/abs/2505.12863) find
  25. Minju Kim, Joonhyeon Bae, Eunsik Shin, and Kyogu Lee. 2025. Synthetic Dataset Generation for String Ensemble Separation. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. find
  26. Junyoung Koh, Soo Yong Kim, Yongwon Choi, and Gyu Hyeong Choi. 2025. Jamendo-QA: A Large-Scale Music Question Answering Dataset. arXiv preprint. ArXiv:2509.15662 [cs]. — [arXiv:2509.15662](https://arxiv.org/abs/2509.15662) find
  27. Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. 2025a. NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples. arXiv preprint. ArXiv:2410.14669 [cs]. find
  28. Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, and 2 others. 2025b. OmniBench: Towards The Future of Un…
  29. Daniel Chenyu Lin, Michael Freeman, and John Thickstun. 2025. Factual and Musical Evaluation Metrics for Music Language Models. arXiv preprint. ArXiv:2511.05550 [cs]. — [arXiv:2511.05550](https://arxiv.org/abs/2511.05550) find
  30. Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. 2023. Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning. arXiv preprint. ArXiv:2308.11276 [cs]. — [arXiv:2308.11276](https://arxiv.org/abs/2308.11276) find
  31. Shansong Liu, Atin Sakkeer Hussain, Qilong Wu, Chenshuo Sun, and Ying Shan. 2024a. M$^{2}$UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models. arXiv preprint. ArXiv:2311.11255 [cs]. — [arXiv:2311.11255](https://arxiv.org/abs/2311.11255) find
  32. Shansong Liu, Atin Sakkeer Hussain, Qilong Wu, Chenshuo Sun, and Ying Shan. 2024b. MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models. arXiv preprint. ArXiv:2412.06660 [cs]. — [arXiv:2412.06660](https://arxiv.org/abs/2412.06660) find
  33. Yinghao Ma, Anders Øland, Anton Ragni, Bleiz Mac-Sen Del Sette, Charalampos Saitis, Chris Donahue, Chenghua Lin, Christos Plachouras, Emmanouil Benetos, Elona Shatri, Fabio Morreale, Ge Zhang, György Fazekas, Gus Xia, Huan Zhang, Ilaria Manco, Jiawen Huang, Julien Guinot, Liwei Lin, and 23 others. 2… — [arXiv:2408.14340](https://arxiv.org/abs/2408.14340) find
  34. Jiří Mayer, Pavel Pecina, and Jan Hajič jr. 2024. Smashcima (2025-03-28). LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL). find
  35. Christopher Small. 1999. Musicking — the meanings of performing and listening. A lecture. Music Education Research, 1(1):9–22. _eprint: https://doi.org/10.1080/1461380990010102. — [doi:10.1080/1461380990010102](https://doi.org/10.1080/1461380990010102) ↗
  36. Jiří Mayer, Pavel Pecina, and Jan Hajič. 2025. Smashcima: Full-Page Handwritten Music Document Synthesizer. In Proceedings of the 12th International Conference on Digital Libraries for Musicology, DLfM ’25, pages 119–123, New York, NY, USA. Association for Computing Machinery. find
  37. Mingni Tang, Jiajia Li, Lu Yang, Zhiqiang Zhang, Jinghao Tian, Zuchao Li, Lefei Zhang, and Ping Wang. 2025. NOTA: Multimodal Music Notation Understanding for Visual Large Language Model. arXiv preprint. ArXiv:2502.14893 [cs]. find
  38. Samuel A. Mehr, Manvir Singh, Dean Knox, Daniel M. Ketter, Daniel Pickens-Jones, S. Atwood, Christopher Lucas, Nori Jacoby, Alena A. Egner, Erin J. Hopkins, Rhea M. Howard, Joshua K. Hartshorne, Mariela V. Jennings, Jan Simson, Constance M. Bainbridge, Steven Pinker, Timothy J. O’Donnell, Max M. Kra… find
  39. Microsoft, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi-ling Chen, Qi Dai, Xiyang Dai, and 56 others. 2025. Phi-4-Mini T…
  40. Gagan Mundada, Yash Vishe, Amit Namburi, Xin Xu, Zachary Novack, Julian McAuley, and Junda Wu. 2025. WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning. arXiv preprint. ArXiv:2509.04744 [cs].
  41. OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, A. J. Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, and 400 others. 2024. GPT-4o…
  42. Charilaos Papaioannou, Emmanouil Benetos, and Alexandros Potamianos. 2025. Universal Music Representations? Evaluating Foundation Models on World Music Corpora. International Society for Music Information Retrieval Conference (ISMIR 2025). find
  43. Conference Name: Ismir 2025 Hybrid Conference. find
  44. Liam Pond and Ichiro Fujinaga. 2025. Teaching LLMs Music Theory with In-Context Learning and Chain-of-Thought Prompting: Pedagogical Strategies for Machines. arXiv preprint. ArXiv:2503.22853 [cs]. — [arXiv:2503.22853](https://arxiv.org/abs/2503.22853) find
  45. Narun Raman, Taylor Lundy, and Kevin Leyton-Brown. 2025. Reasoning Models are Test Exploiters: Rethinking Multiple-Choice. arXiv preprint. ArXiv:2507.15337 [cs] version: 1. — [arXiv:2507.15337](https://arxiv.org/abs/2507.15337) find
  46. Zhilin Wang, Zhe Yang, Yun Luo, Yafu Li, Xiaoye Qu, Ziqian Qiao, Haoran Zhang, Runzhe Zhan, Derek F. Wong, Jizhe Zhou, and Yu Cheng. 2025. Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning. arXiv preprint. ArXiv:2509.04059 [cs]. — [arXiv:2509.04059](https://arxiv.org/abs/2509.04059) find
  47. Benno Weck, Ilaria Manco, Emmanouil Benetos, Elio Quinton, George Fazekas, and Dmitry Bogdanov. 2024. MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models. arXiv preprint. ArXiv:2408.01337 [cs]. — [arXiv:2408.01337](https://arxiv.org/abs/2408.01337) find
  48. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025. Qwen3-Omni Technical Report. arXiv preprint. ArXiv:2509.17765…
  49. Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Ziyang Ma, Liumeng Xue, Ziyu Wang, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, and 16 others. 2024. ChatMusician: Understanding and Generating Music Intrinsi… find
  50. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024. MMMU: A Massive Multi-discipline Multimodal U…
  51. Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. 2025. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. arXiv preprint. ArXiv:2409.02813 [cs]. find
  52. Yongyi Zang, Sean O’Brien, Taylor Berg-Kirkpatrick, Julian McAuley, and Zachary Novack. 2025. Are you really listening? Boosting Perceptual Awareness in Music-QA Benchmarks. Proceedings of the 26th International Society for Music Information Retrieval Conference (ISMIR 2025). find
  53. report baseline performance, at least random choice baseline Jiahao Zhao, Yunjia Li, Wei Li, and Kazuyoshi Yoshii. 2025. ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following. arXiv preprint. ArXiv:2509.23350 [cs]. — [arXiv:2509.23350](https://arxiv.org/abs/2509.23350) find
  54. be released openly, and all details must be released: all decisions, evaluation protocol, etc. find
  55. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36:46595–4…
  56. be prevented from leakage to training data of LLMs (Balloccu et al., 2024) and from over-evaluating on the test set (which is problematic as it may goe directly against the fully-open benchmarks) find
  57. Adam Štefunko, Suhit Chiruthapudi, Carlos Eduardo Cancino-Chacón, and jr. Jan Hajič. 2025. Basso continuo goes digital: Collecting and aligning a symbolic dataset of continuo performance. In The AI Music Creativity Conference (AIMC), pages 1–16, Brussels, Belgium. Vrije Universiteit Brussel. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sourada-2026-thesis-proposal-multimodal-benchmark-music/