Elamur Research⌕
☰
home / papers / 2025 / WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning

WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning

Paper · 2025 · Gagan Mundada, Yash Vishe, Amit Namburi et al.

Summary

Multimodal LLMs perform well on many vision-language tasks, but it is unclear whether they can reason about sheet music. Earlier music benchmarks either turned scores into text for unimodal LLMs and used textbook-style questions, or focused on transcription and generation (MusicNet, MAESTRO). WildScore fills this gap. It pairs images of real scores by actual composers with questions and discussions that users posted on Reddit. A two-level taxonomy organizes the questions into five broad areas (Harmony & Tonality, Rhythm & Meter, Texture, Expression & Performance, Form), each split into finer subcategories. Because forum questions are open-ended and often subjective, an LLM rewrites each post as a multiple-choice question, with answer options drawn from the thread's annotated answer. This allows automatic scoring. When the authors tested current MLLMs, accuracy was uneven across topics. Models handled surface-level recognition reasonably well but struggled with deeper abstraction, complex rhythm and orchestration. Diagnostic tests suggest that many errors come from misreading the notation itself. WildScore offers a realistic testbed for progress in visual-symbolic music understanding.

Key points

  • WildScore is released on Hugging Face (GM77/WildScore), with code on GitHub (GaganVM/WildScore), under CC BY-NC 4.0. Reddit usernames and other metadata were removed before release.
  • Several models were evaluated with and without the score image, including Gemma-3, MiniCPM, InternVL, LLaVA and GPT-4.1-mini. Adding the image did not help consistently: average accuracy improved for Gemma-3 but dropped for MiniCPM, InternVL and LLaVA.
  • In a perception-only probe of 50 score images with simple factual questions (e.g., clef identification, counting symbols), GPT-4.1-mini scored 52%, InternVL 38% and LLaVA 26%.
  • When asked to convert score images into ABC notation, InternVL and LLaVA often produced invalid or looping output. GPT-4.1-mini handled short single-staff excerpts but degraded on longer, denser passages.
  • The authors note limitations of the Reddit source: a bias toward mainstream topics, occasional toxic language, and answers that are sometimes technically inaccurate.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California San DiegoSan Diego · United States73 works · 51,948 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Gagan Mundada, Yash Vishe, Amit Namburi, Xin Xu, Zachary Novack, Julian McAuley and Junda Wu. 2025. WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.853

@inproceedings{mundada2025wildscore,
  title = {{WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning}},
  author = {Gagan Mundada, Yash Vishe, Amit Namburi, Xin Xu, Zachary Novack, Julian McAuley and Junda Wu},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.853},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.853}
}

References

57 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.
  2. Audiveris. 2025. Audiveris. https://github.com/ Audiveris/audiveris. Version 5.6.2. Released 2025-07-18. Accessed: 2025-08-31. ↗
  3. Jinze Bai and 1 others. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966.
  4. Kang Chen and Xiangqian Wu. 2024. Vtqa: Visual text question answering via entity alignment and cross-media reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27218–27227. find
  5. Zhe Chen and 1 others. 2023. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238.
  6. Louis Couturier, Louis Bigo, Florence Levé, and Markus Neuwirth. 2022. A dataset of symbolic texture annotations in mozart piano sonatas. In Proceedings of the 23rd International Society for Music Information Retrieval Conference (ISMIR 2022), pages 509–516. find
  7. Lukasz Czajka, Malihe Alikhani, Patrick Verga, and Yonatan Belinkov. 2024a. Musical understanding benchmark: A challenge for large language models. arXiv preprint arXiv:2410.02084. — [arXiv:2410.02084](https://arxiv.org/abs/2410.02084) find
  8. Lukasz Czajka, Malihe Alikhani, Patrick Verga, and Yonatan Belinkov. 2024b. Musictheory-bench. ArXiv preprint arXiv:2410.02084. — [arXiv:2410.02084](https://arxiv.org/abs/2410.02084) find
  9. W. Bas de Haas and Anja Volk. 2016. Meter detection in symbolic music using inner metric analysis. In Proceedings of the 17th International Society for Music Information Retrieval Conference (ISMIR 2016), pages 441–447. find
  10. Chengkai Huang, Junda Wu, Yu Xia, Zixu Yu, Ruhan Wang, Tong Yu, Ruiyi Zhang, Ryan A Rossi, Branislav Kveton, Dongruo Zhou, and 1 others. 2025. Towards agentic recommender systems in the era of multimodal large language models. arXiv preprint arXiv:2503.16734. — [arXiv:2503.16734](https://arxiv.org/abs/2503.16734) find
  11. Hsiao-Tzu Hung, Joann Ching, Seungheon Doh, Nabin Kim, Juhan Nam, and Yi-Hsuan Yang. 2021. Emopia: A multi-modal pop piano dataset for emotion recognition and emotion-based music generation. Preprint, arXiv:2108.01374. — [arXiv:2108.01374](https://arxiv.org/abs/2108.01374) find
  12. Md Farhan Ishmam, Ishmam Tashdeed, Talukder Asir Saadat, Md Hamjajul Ashmafee, Abu Raihan Mostofa Kamal, and Md Azam Hossain. 2025. Visual robustness benchmark for visual question answering (vqa). In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6623–6633. IEEE. find
  13. Maximos Kaliakatsos-Papakostas, Dimos Makris, Konstantinos Soiledis, Konstantinos-Theodoros Tsamis, Vassilis Katsouros, and Emilios Cambouropoulos. 2025. Harmonytok: Comparing methods for harmony tokenization for machine learning. Preprints. find
  14. Rahima Khanam and Muhammad Hussain. 2024. Yolov11: An overview of the key architectural enhancements. arXiv preprint arXiv:2410.17725. — [arXiv:2410.17725](https://arxiv.org/abs/2410.17725) find
  15. Chris Donahue, Zachary C. Lipton, and Julian McAuley. 2018. The nes music database: A multi-instrumental dataset with expressive performance attributes. In ISMIR. find
  16. Junnan Li, Dongxu Li, and 1 others. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597.
  17. Brad Dwyer, Joseph Nelson, Tyler Hansen, and 1 others. 2025. Roboflow (version 1.0). Computer vision software. find
  18. Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. Microsoft coco: Common objects in context. Preprint, arXiv:1405.0312.
  19. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, and 1 others. 2024. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075.
  20. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. Preprint, arXiv:1612.00837.
  21. Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse H. Engel, and Douglas Eck. 2019. Enabling factorized piano music modeling and generation with the MAESTRO dataset. In 7th International Conference on Learning Representations, ICLR 20… find
  22. Shengding Hu and 1 others. 2024. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395.
  23. Yuheng Lin, Zheqi Dai, and Qiuqiang Kong. 2024. Musicscore: A dataset for music score modeling and generation. arXiv preprint arXiv:2406.11462. — [arXiv:2406.11462](https://arxiv.org/abs/2406.11462) find
  24. Haotian Liu, Chunyuan Zhang, and 1 others. 2023a. Visual instruction tuning. arXiv preprint arXiv:2304.08485.
  25. Haotian Liu and 1 others. 2023b. Visual instruction tuning. arXiv preprint arXiv:2304.08485.
  26. Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. 2024. Layoutllm: Layout instruction tuning with large language models for document understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15630–15640. find
  27. Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, Kai Li, Keliang Li, Siyou Li, Xinfeng Li, Xiquan Li, Zheng Lian, Yuzhe Liang, Minghao Liu, Zhikang Niu, and 15 others. 2025. Mmar: A challenging benchmark for deep reasoning in spe… — [arXiv:2505.13032](https://arxiv.org/abs/2505.13032) find
  28. Musitek Corporation. 2024. Smartscore 64 ne pro edition. https://www.musitek.com/ smartscore64-pro.html. Accessed: 2025-08-31. ↗
  29. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  30. Pavel Pecina and 1 others. 2017. In search of a dataset for handwritten optical music recognition: Introducing muscima++. arXiv preprint arXiv:1703.04824. — [arXiv:1703.04824](https://arxiv.org/abs/1703.04824) find
  31. Colin Raffel. 2016. Learning-based methods for comparing sequences, with applications to audio-to-midi alignment and matching. PhD dissertation, Columbia University. find
  32. Ana Rebelo, João Pimentel, Jaime Cardoso, and 1 others. 2012. Optical music recognition: state-of-the-art and open issues. International Journal of Multimedia Information Retrieval, 1(3):173–190. find
  33. Reddit. 2024. Reddit. https://www.reddit.com/. Accessed: 2025-08-31. ↗
  34. Ruoyu Wang, Tong Yu, Junda Wu, Yao Liu, Julian McAuley, and Lina Yao. 2025. Weakly-supervised vlm-guided partial contrastive learning for visual language navigation. arXiv preprint arXiv:2506.15757. — [arXiv:2506.15757](https://arxiv.org/abs/2506.15757) find
  35. Junda Wu, Jessica Echterhoff, Kyungtae Han, Amr Abdelraouf, Rohit Gupta, and Julian McAuley. 2025a. Pdb-eval: An evaluation of large multimodal models for description and explanation of personalized driving behavior. In 2025 IEEE Intelligent Vehicles Symposium (IV), pages 242–248. IEEE. find
  36. Junda Wu, Warren Li, Zachary Novack, Amit Namburi, Carol Chen, and Julian McAuley. 2025b. Col-lap: Contrastive long-form language-audio pretraining with musical temporal structure augmentation. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pag… find
  37. Junda Wu, Xintong Li, Tong Yu, Yu Wang, Xiang Chen, Jiuxiang Gu, Lina Yao, Jingbo Shang, and Julian McAuley. 2024a. Commit: Coordinated instruction tuning for multimodal large language models. arXiv preprint arXiv:2407.20454. — [arXiv:2407.20454](https://arxiv.org/abs/2407.20454) find
  38. S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2024. Mmau: A massive multi-task audio understanding and reasoning benchmark. Preprint, arXiv:2410.19168. find
  39. Junda Wu, Hanjia Lyu, Yu Xia, Zhehao Zhang, Joe Barrow, Ishita Kumar, Mehrnoosh Mirtaheri, Hongjie Chen, Ryan A Rossi, Franck Dernoncourt, and 1 others. 2024b. Personalized multimodal large language models: A survey. arXiv preprint arXiv:2412.02142. — [arXiv:2412.02142](https://arxiv.org/abs/2412.02142)
  40. Rohan Surana, Aakash Varshney, and Vishnu Pendyala. 2022. Deep learning for conversions between melodic frameworks of indian classical music. In Proceedings of Second International Conference on Advances in Computer Engineering and Communication Systems, pages 1–12, Singapore. Springer Nature Singap… find
  41. Junda Wu, Zachary Novack, Amit Namburi, Hao-Wen Dong, Carol Chen, Jiaheng Dai, and Julian McAuley. 2025c. Futga-mir: Enhancing fine-grained and temporally-aware music understanding with music information retrieval. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Pro… find
  42. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  43. Junda Wu, Yu Xia, Tong Yu, Xiang Chen, Sai Sree Harsha, Akash V Maharaj, Ruiyi Zhang, Victor Bursztyn, Sungchul Kim, Ryan A Rossi, and 1 others. 2025d. Doc-react: Multi-page heterogeneous document question-answering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Ling…
  44. John Thickstun, Zaid Harchaoui, and Sham M. Kakade. 2017a. Learning features of music from scratch. In International Conference on Learning Representations (ICLR). find
  45. John Thickstun, Zaid Harchaoui, and Sham M Kakade. 2017b. Learning features of music from scratch. arXiv preprint arXiv:1611.09827. — [arXiv:1611.09827](https://arxiv.org/abs/1611.09827) find
  46. Lukas Tuggener, Ismail Elezi, Jurgen Schmidhuber, Marcello Pelillo, and Thilo Stadelmann. 2018. Deepscores-a dataset for segmentation, detection and classification of tiny objects. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 3704–3709. IEEE. find
  47. Dimitri von Rütte, Luca Biggio, Yannic Kilcher, and Thomas Hofmann. 2022. Figaro: Generating symbolic music with fine-grained artistic control. arXiv preprint arXiv:2201.10936. — [arXiv:2201.10936](https://arxiv.org/abs/2201.10936) find
  48. Junda Wu, Yuxin Xiong, Xintong Li, Yu Xia, Ruoyu Wang, Yu Wang, Tong Yu, Sungchul Kim, Ryan A Rossi, Lina Yao, and 1 others. 2025e. Mitigating visual knowledge forgetting in mllm instruction-tuning via modality-decoupled gradient descent. arXiv preprint arXiv:2502.11740. find
  49. Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A Rossi, Ruiyi Zhang, and 1 others. 2024c. Visual prompting in multimodal large language models: A survey. arXiv preprint arXiv:2409.15310. — [arXiv:2409.15310](https://arxiv.org/abs/2409.15310)
  50. Gus Xia. 2016. Expressive Collaborative Music Performance via Machine Learning. Ph.D. thesis, Carnegie Mellon University. find
  51. Weihan Xu, Julian McAuley, Taylor Berg-Kirkpatrick, Shlomo Dubnov, and Hao-Wen Dong. 2024. Generating symbolic music from natural language prompts using an llm-enhanced dataset. Preprint, arXiv:2410.02084. — [arXiv:2410.02084](https://arxiv.org/abs/2410.02084) find
  52. An Yan, Zhengyuan Yang, Junda Wu, Wanrong Zhu, Jianwei Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Julian McAuley, Jianfeng Gao, and 1 others. 2024. List items one by one: A new data source and learning paradigm for multimodal llms. arXiv preprint arXiv:2404.16375. — [arXiv:2404.16375](https://arxiv.org/abs/2404.16375) find
  53. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. 2024. Air-bench: Benchmarking large audio-language models via generative comprehension. Preprint, arXiv:2402.07729.
  54. Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490.
  55. Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, and 1 others. 2024. Chatmusician: Understanding and generating music intrinsically with llms. arXiv preprint arXiv:2402.16153. — [arXiv:2402.16153](https://arxiv.org/abs/2402.16153) find
  56. Yongyi Zang, Sean O’Brien, Taylor Berg-Kirkpatrick, Julian McAuley, and Zachary Novack. 2025. Are you really listening? boosting perceptual awareness in music-qa benchmarks. arXiv preprint arXiv:2504.00369. — [arXiv:2504.00369](https://arxiv.org/abs/2504.00369) find
  57. Fengbin Zhu, Ziyang Liu, Xiang Yao Ng, Haohui Wu, Wenjie Wang, Fuli Feng, Chao Wang, Huanbo Luan, and Tat Seng Chua. 2024. Mmdocbench: Benchmarking large vision-language models for fine-grained visual document understanding. arXiv preprint arXiv:2410.21311. — [arXiv:2410.21311](https://arxiv.org/abs/2410.21311) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mundada-2025-wildscore-benchmarking-mllms-wild-symbolic/