Elamur Research⌕
☰
home / papers / 2026 / ORCA: Open-ended Response Correctness Assessment for Audio Question Answering

ORCA: Open-ended Response Correctness Assessment for Audio Question Answering

Paper · 2026 · Šimon Sedláček, Sara Barahona, Cecilia Bolaños et al.

Summary

Large audio language models (LALMs) are usually tested with multiple-choice questions, even though free-form answers better reflect real use. Grading free-form answers is hard: correct responses can be phrased in many ways, partial credit is often needed, and human raters frequently disagree, especially on subjective audio content. The authors built a three-stage annotation pipeline. First, Gemini-2.5-Flash produces textual rationales and 15 LALMs produce candidate answers. Next, humans assign 1–5 correctness scores and flag problems. Finally, humans correct the flagged items with AI help. On this data they trained ORCA, a small text-only scorer. Instead of a single score, ORCA predicts the full distribution of human ratings (Bernoulli, Beta or Multinomial), using a curriculum that moves from synthetic data to LLM-judge labels to human labels. ORCA tracks average human judgments closely on both seen and unseen benchmarks and beats LLM judges, including Gemini 2.5 Flash, while needing only one forward pass. Its predicted variance also signals where humans disagree and which benchmark items are flawed. This makes open-ended evaluation of audio models cheaper and more trustworthy.

Key points

  • The dataset contains 9,663 human annotations of 3,699 question-answer pairs from 15 LALMs on MMAU, MMAR and MMAU-Pro, with Krippendorff's α of about 0.82 after filtering.
  • ORCA (e.g., built on Llama3.2-3B) reaches a Spearman correlation of 0.91 with mean human ratings on seen benchmarks and 0.85 on unseen ones.
  • Predicted variance correlates significantly with inter-annotator variance for 7 of 8 held-out LALMs (ρ between 0.22 and 0.74).
  • On 248 clean and 136 flagged MMAU-Pro questions, flagged items receive higher predicted variance (Mann-Whitney p < 0.0001). Median variance rises from 0.38 to 0.59 with the Multinomial model and to 0.57 with the Beta model.
  • The trained models, code, the annotation set and corrected benchmark versions are released at github.com/BUTSpeechFIT/ORCA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations🎓Brno University of TechnologyBrno · Czechia11 works · 32,326 citations🎓Tufts UniversityMedford · United States6 works · 402 citations🎓Universidad Autónoma de MadridMadrid · Spain7 works · 21 citations🎓Universidad de Buenos AiresBuenos Aires · Argentina8 works · 447 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Question answering · 441 worksSpeech & spoken language · 480 works

Cite

Cite as Šimon Sedláček, Sara Barahona, Cecilia Bolaños, Laura Herrera-Alarcón, Sathvik Udupa, Fernando López, Allison Ferner, Bolaji Yusuf, Alicia Lozano-Diez, Santosh Kesiraju, Ramani Duraiswami and Jan Černocký. 2026. ORCA: Open-ended Response Correctness Assessment for Audio Question Answering. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.798

@article{sedlacek2026orca,
  title = {{ORCA: Open-ended Response Correctness Assessment for Audio Question Answering}},
  author = {Šimon Sedláček, Sara Barahona, Cecilia Bolaños, Laura Herrera-Alarcón, Sathvik Udupa, Fernando López, Allison Ferner, Bolaji Yusuf, Alicia Lozano-Diez, Santosh Kesiraju, Ramani Duraiswami and Jan Černocký},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.798},
  url = {https://doi.org/10.1162/tacl.a.798}
}

References

46 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. 2025. Which of these best describes multiple choice evaluation with LLMs? a) forced B) flawed C) fixable D) all of the above. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper…
  2. Debarpan Bhattacharya, Apoorva Kulkarni, and Sriram Ganapathy. 2025. Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning. In Interspeech 2025, pages 2068–2072. find
  3. Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Börschinger, and Tal Schuster. 2022. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 291–305, Abu Dha…
  4. Khaoula Chehbouni, Mohammed Haddou, Jackie CK Cheung, and Golnoosh Farnadi. 2025. Neither Valid nor Reliable? Investigating the Use of LLMs as Judges. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems Position Paper Track. find
  5. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. Qwen2-Audio Technical Report.
  6. Alexander P. Dawid and Allan M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20–28. find
  7. Ding Ding, Zeqian Ju, and Yichong Leng. 2025. Kimi-Audio Technical Report. Technical report, MoonshotAI.
  8. Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disagreement for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6715–6726, Singapore. Association for Computational Linguistics. find
  9. Eve Fleisig, Su Lin Blodgett, Dan Klein, and Zeerak Talat. 2024. The perspectivist paradigm shift: Assumptions and challenges of capturing human labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (… find
  10. Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin…
  11. Gemini Team. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities.
  12. Gemma Team. 2025a. Gemma 3 Technical Report. find
  13. Gemma Team. 2025b. Gemma 3n. find
  14. Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S. Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro. 2025. Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities. In Proceedings of the 42 nd International Conference on Machine… find
  15. Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024. GAMA: A large audio-language model with advanced audio understanding and complex reasoning abilities. In Proceedings of the 2024 Conference on Empiric…
  16. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Metho…
  17. Patrizio Giovannotti. 2023. Evaluating Machine Translation Quality with Conformal Predictive Distributions. In Proceedings of the Twelfth Symposium on Conformal and Probabilistic Prediction with Applications, volume 204 of Proceedings of Machine Learning Research, pages 413–429. PMLR. find
  18. Klaus Krippendorff. 2019. Content Analysis: An Introduction to Its Methodology. SAGE Publications, Inc. find
  19. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130, Atlanta, Georgia. Associati…
  20. Sonal Kumar, Šimon Sedláček, Vaibhavi Lokegaonkar, Fernando López, Wenyi Yu, Nishit Anand, Hyeonggon Ryu, Lichang Chen, Maxim Plička, Miroslav Hlaváček, William Fineas Ellingwood, Sathvik Udupa, Siyuan Hou, Allison Ferner, Sara Barahona, Cecilia Bolaños, Satish Rahi, Laura Herrera-Alarcón, Satvik… find
  21. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR. OpenReview.net.
  22. Yichong Leng, Xu Tan, Sheng Zhao, Frank Soong, Xiang-Yang Li, and Tao Qin. 2021. MBNET: MOS Prediction for Synthesized Speech with Mean-Bias Network. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 391–395. IEEE. find
  23. Wen-Chin Huang, Erica Cooper, Junichi Yamagishi, and Tomoki Toda. 2022. LDNet: Unified Listener Dependent Modeling in MOS Prediction for Synthetic Speech. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 896–900. IEEE. find
  24. Elisa Leonardelli, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, Alexandra Uma, and Massimo Poesio. 2023. SemEval-2023 task 11: Learning with disagreements (LeWiDi). In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval…
  25. Llama Team. 2024. The Llama 3 Herd of Models.
  26. Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, and Hung-Yi Lee. 2025. Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data. In IEEE International Conference on Acoustics, Speech and Signal Proce… find
  27. Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Chih-Kai Yang, Chee-En Yu, Chun-Wei Chen, Wei-Chih Chen, Chien-yu Huang, Yi-Cheng Lin, Yu-Xiang Lin, Chi-An Fu, Chun-Yi Kuan, Wenze Ren, Xuanjun Chen, Wei-Ping Huang, En-Pei Hu, Tzu-Quan Lin, Yuan-Kuei Wu, Kuan-Po Huang, Hsiao… find
  28. Fernando López, Santosh Kesiraju, and Jordi Luque. 2025. Robustness assessment of large audio language models in multiple-choice evaluation. find
  29. Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, Kai Li, Keliang Li, Siyou Li, Xinfeng Li, Xiquan Li, Zheng Lian, Yuzhe Liang, Minghao Liu, Zhikang Niu, Tianrui Wang, Yuping Wang, Yuxuan Wang, Yihao Wu, Guanrou Yang, Jianwei Yu,… find
  30. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority Grigor Nalbandyan, Rima Shahbazyan, and Evelina Bakhturina. 2025. SCORE: Systematic COnsistency and robustness evaluation for large language models. In Proceedings of the 20… find
  31. Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Alexander Miserlis Hoyle, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, and Elliott Ash. 2026. Can reasoning help large language models capture human annotator disagreement? In Proceedings of the 19th Conference of the European Chapter of the Associatio… find
  32. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671– 10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  33. Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfe… find
  34. S. Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2024. MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark. In The Twelfth International Conference on Learning Representations. 4316, Albu… find
  35. Marta Sandri, Elisa Leonardelli, Sara Tonelli, and Elisabetta Jezek. 2023. Why don’t you do it right? analysing annotators’ disagreement in subjective tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2428–2441, Dubrovnik, Cr… find
  36. Dingdong Wang, Junan Li, Jincenzi Wu, Dongchao Yang, Xueyuan Chen, Tianhua Zhang, and Helen M. Meng. 2026. MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark. In The Fourteenth International Conference on Learning Representations. find
  37. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. SALMONN: Towards Generic Hearing Abilities for Large Language Models. In The Twelfth International Conference on Learning Representations.
  38. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2022. Learning from Disagreement: A Survey. Journal of Artificial Intelligence Research, 72:1385–1470. find
  39. Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, Michal Guerqu… find
  40. Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F. Chen. 2025. AudioBench: A universal benchmark for audio large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational L…
  41. Wen Wu, Chao Zhang, and Philip Woodland. 2023. Estimating the uncertainty in emotion attributes using deep evidential regression. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15681–15695, Toronto, Canada. Association for Co… find
  42. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen2.5-Omni Technical Report.
  43. Chih-Kai Yang, Neo Ho, Yen-Ting Piao, and Hung yi Lee. 2025. SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information. In Interspeech, pages 1788–1792. ISCA. find
  44. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. 2024. AIR-bench: Benchmarking large audio-language models via generative comprehension. In Proceedings of the 62nd Annual Meeting of the Association for Computati…
  45. Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 2024. GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot.
  46. Xie Zhifei, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. 2025. Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 23829–23851, Suzhou, China. Association…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sedlacek-2026-orca-open-ended-response-correctness/