Elamur Research⌕
☰
home / papers / 2026 / HumMusQA: A Human-written Music Understanding QA Benchmark Dataset

HumMusQA: A Human-written Music Understanding QA Benchmark Dataset

Paper · 2026 · Benno Weck, Pablo Puentes, Andrea Poltronieri et al.

Summary

Many existing music QA benchmarks for Large Audio-Language Models (LALMs) were built by having LLMs rewrite captions or tags. As a result, text-only models can often solve them without listening, so the benchmarks reward language priors more than audio perception. HumMusQA addresses this with a small, carefully curated benchmark. Three music theory experts wrote 320 four-option multiple-choice questions. Their design drew on ABRSM and GCSE curricula and on earlier benchmarks such as MuChoMusic and MMAU. The questions refer to excerpts from Creative Commons tracks on Jamendo. Each question went through blind cross-answering and repeated revision until no reviewer disagreed. GPT-5 then labelled the questions with a 13-category taxonomy and three difficulty levels, and two experts checked those labels.

The authors evaluated six LALMs, shuffling the answer options across runs. Qwen2.5-Omni-7B scored best. All models handled contextual and cultural questions better than theory-heavy ones, and accuracy fell as difficulty increased. With noise or silence instead of audio, models still beat chance, which shows that text cues remain exploitable. Because both the audio and the questions are openly licensed, evaluation is fully reproducible.

Key points

  • Dataset: 320 expert-written questions over 108 Jamendo tracks, with excerpts of 30–90 s. It is released under Creative Commons on Zenodo (doi:10.5281/zenodo.18462524).
  • Labels:
    • Categories follow an adapted MuChoMusic taxonomy of 13 categories. The most frequent are Instrumentation (19.7%), Harmony (11.3%) and Melody (10.6%).
    • Difficulty splits into low (44.4%), medium (38.4%) and high (17.2%).
  • Models tested: gemini-2.5-flash, gpt-audio, audio-flamingo-3, qwen2.5-omni-7b, audsemthinker and music-flamingo.
  • Protocol: each question was run four times with shuffled options. gemini-2.5-flash parsed the model answers.
  • Results:
    • qwen2.5-omni-7b scored highest (64.3 ± 2.9%) and was also the most stable under option shuffling.
    • With Gaussian noise or silence instead of real audio, accuracy fell to roughly 38–47%, which is still above the 25% chance level.
    • In a text-only setting, gemini-2.5-flash answered about half of the high-difficulty questions correctly. The authors trace this to cues in question wording and to unevenly plausible distractors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universitat Pompeu FabraBarcelona · Spain48 works · 12,092 citations🎓Universitat Autònoma de BarcelonaCerdanyola del Vallès · Spain5 works · 9,880 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksSignal Processing · 493 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 487 works

Cite

Cite as Benno Weck, Pablo Puentes, Andrea Poltronieri, Satyajeet Prabhu and Dmitry Bogdanov. 2026. HumMusQA: A Human-written Music Understanding QA Benchmark Dataset. Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.nlp4musa-1.9

@inproceedings{weck2026hummusqa,
  title = {{HumMusQA: A Human-written Music Understanding QA Benchmark Dataset}},
  author = {Benno Weck, Pablo Puentes, Andrea Poltronieri, Satyajeet Prabhu and Dmitry Bogdanov},
  booktitle = {Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.nlp4musa-1.9},
  url = {https://doi.org/10.18653/v1/2026.nlp4musa-1.9}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. 2023. MusicLM: Generating Music From Text. arXiv preprint arXiv:2301.11325. — [arXiv:2301.11325](https://arxiv.org/abs/2301.11325) find
  2. Pablo Alonso-Jiménez, Pedro Ramoneda, Recep Oguz Araz, Andrea Poltronieri, and Dmitry Bogdanov. 2025. OMAR-RQ: open music audio representation model trained with multi-feature masked token prediction. In Proceedings of the 33rd ACM International Conference on Multimedia, MM 2025, Dublin, Ireland, Oc… find
  3. Roser Batlle-Roca, Laura Ibáñez-Martínez, Xavier Serra, Emilia Gómez, and Martín Rocamora. 2025. MusGO: A Community-Driven Framework for Assessing Openness in Music-Generative AI. In Proceedings of the 26th International Society for Music Information Retrieval Conference, ISMIR 2025, Daejeon, South… find
  4. Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. 2019. The MTG-Jamendo Dataset for Automatic Music Tagging. In Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML 2019), Long Beach, CA, United States. find
  5. Udupa, Siyuan Hou, Allison Ferner, Sara Barahona, Cecilia Bolaños, Satish Rahi, Laura Herrera-Alarcón, Satvik Dixit, Rupali S. Patil, Soham Deshmukh, Lasha Koroshinadze, Yao Liu, Leibny Paola Garcia Perera, Eleni Zanou, Themos Stafylakis, Joon Son Chung, David Harwath, Chao Zhang, Dinesh Manocha, Al… — [arXiv:2508.13992](https://arxiv.org/abs/2508.13992) find
  6. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models. arXiv preprint. ArXiv:2311.07919 [cs, eess].
  7. Zihao Deng, Yinghao Ma, Yudong Liu, Rongchen Guo, Ge Zhang, Wenhu Chen, Wenhao Huang, and Emmanouil Benetos. 2024. MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response. In Findings of the Association for Computational Linguistics: NAACL 2024, pa…
  8. Edith Law, Kris West, Michael I. Mandel, Mert Bay, and J. Stephen Downie. 2009. Evaluation of Algorithms Using Games: The Case of Music Tagging. In Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009, pages 387–392, Kobe International Conference Cente… find
  9. Sreyan Ghosh, Arushi Goel, Lasha Koroshinadze, Sang-gil Lee, Zhifeng Kong, Joao Felipe Santos, Ramani Duraiswami, Dinesh Manocha, Wei Ping, Mohammad Shoeybi, and Bryan Catanzaro. 2025a. Music Flamingo: Scaling Music Understanding in Audio Language Models. arXiv preprint. Version Number: 1. find
  10. Yu-Xiang Lin, Chen-An Li, Sheng-Lun Wei, Po-Chun Chen, Hsin-Hsi Chen, and Hung-yi Lee. 2025. Hearing the Order: Investigating Selection Bias in Large Audio-Language Models. CoRR, abs/2510.00628. ArXiv: 2510.00628. — [arXiv:2510.00628](https://arxiv.org/abs/2510.00628) find
  11. Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro. 2025b. Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities. In Forty-second International Conference on Machine Learning. find
  12. Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun, and Ying Shan. 2024. Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 286–290, Seoul,… find
  13. Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. 2025. Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models. arXiv preprint arXiv. find
  14. Ilaria Manco, Benno Weck, Seungheon Doh, Minz Won, Yixiao Zhang, Dmitry Bogdanov, Yusong Wu, Ke Chen, Philip Tovstogan, Emmanouil Benetos, Elio Quinton, György Fazekas, and Juhan Nam. 2023. The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation. In Machine Learning… find
  15. Gemini Team Google. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv preprint. Version Number: 5.
  16. OpenAI. 2025a. GPT-5 System Card. find
  17. OpenAI. 2025b. gpt-audio Model \| OpenAI API. find
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  19. Laya Balaji Iyer. 2025. Analyzing Audio Understanding in Multimodal LLMs: A Benchmark Grounded in Assistive and Industrial Use Cases. find
  20. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. I…
  21. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, Mexico City, Mexico. Association for Computational Linguistics.
  22. Sonal Kumar, Simon Sedlácek, Vaibhavi Lokegaonkar, Fernando López, Wenyi Yu, Nishit Anand, Hyeonggon Ryu, Lichang Chen, Maxim Plicka, Miroslav Hlavácek, William Fineas Ellingwood, Sathvik S. Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sre… find
  23. MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark. In The Thirteenth International Conference on Learning Representations. pages 6252–6271, Bangkok, Thailand. Association for Computational Linguistics.
  24. Yongyi Zang, Sean O’Brien, Taylor Berg-Kirkpatrick, Julian J. McAuley, and Zachary Novack. 2025. Are You Really Listening? Boosting Perceptual Awareness in Music-QA Benchmarks. In Proceedings of the 26th International Society for Music Information Retrieval Conference, ISMIR 2025, Daejeon, South Kor… find
  25. Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. In 20th Annual Conference of the International Speech Communication Association, Interspeech 2019, Graz, Austria, September 15-19, 2019, pages 3465–3469. ISCA. find
  26. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large Language Models Are Not Robust Multiple Choice Selectors. In The Twelfth International Conference on Learning Representations. find
  27. Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F. Chen. 2025. AudioBench: A Universal Benchmark for Audio Large Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational L…
  28. Benno Weck, Ilaria Manco, Emmanouil Benetos, Elio Quinton, George Fazekas, and Dmitry Bogdanov. 2024. MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models. In Proceedings of the 25th International Society for Music Information Retrieval Conference, ISMIR 2024, pages 825–833… find
  29. Gijs Wijngaard, Elia Formisano, Michele Esposito, and Michel Dumontier. 2025. AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of Sound. find
  30. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen2.5-Omni Technical Report. arXiv preprint. Version Number: 1.
  31. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. 2024. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension. In Proceedings of the 62nd Annual Meeting of the Association for Computati…
  32. Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Liumeng Xue, Ziyang Ma, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, Ruibo Liu, Zili Wang, Chenghua Lin, Qifeng Liu, Tao Jiang, Wenhao Huang, Wenhu Chen, Jie… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/weck-2026-hummusqa-human-written-music-understanding/