Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models
Paper · 2025 · Kuofeng Gao, Shu-Tao Xia, Ke Xu et al.
Summary
Large Audio-Language Models (LALMs) such as GPT-4o can now hold spoken conversations directly with users. Until this work, however, no benchmark tested how well they understand open-ended audio dialogue. Earlier evaluations covered basic audio tasks, audio QA driven by text instructions, or general-purpose chat only.
The authors built ADU-Bench, a collection of four datasets with 20,715 spoken queries. More than 8,000 are real recordings and the rest are synthesized. The datasets cover:
- general conversation,
- domain skills,
- multilingual use,
- ambiguity, a first-of-its-kind set where the same words carry different intent depending on intonation, pauses, homophones or repetition.
Model replies are converted to text and scored by GPT-4 against reference answers. Each reply is scored twice, with the reference and response positions swapped, to cancel order bias. LLaMA-3-70B-Instruct and Qwen-2-72B-Instruct serve as additional judges.
Across 16 LALMs, the main weaknesses were:
- math and code involving formulas,
- common sense and roleplay,
- non-Indo-European languages,
- phonetically driven ambiguity.
The benchmark gives a standardized way to compare LALMs and to identify where spoken-dialogue systems need work.
Key points
- Dataset sizes: ADU-General has 12,000 items (helpful questions, daily questions, daily statements). ADU-Skill has 3,725 items over 12 skills. ADU-Multilingual has 3,600 items in 9 languages. ADU-Ambiguity has 1,390 items in 4 ambiguity types, drawn from phonetics/phonology books.
- Sources: Alpaca, NQ-Bench, WebGLM, Slue HVB, Common Voice, GSM8K, MATH, MBPP, MMLU, HotpotQA, StrategyQA, among others.
- Judge validation: GPT-4 judgments matched human pairwise preferences in over 85% of cases. Swapping positions showed real order bias, particularly when the response appeared before the reference.
- Ambiguity results: Even GPT-4o averaged only 5.22 on pause-based and 6.05 on homophone-based ambiguity. It tended to list several possible readings rather than identify the intended one.
- Ablations:
- Real and synthetic audio gave similar scores on a 1,000-sample comparison.
- Larger models scored better on average but sometimes lost to smaller ones. For example, SALMONN-7B beat SALMONN-13B on Code.
Institutions
China333 works · 87,019 citations🎓University of OxfordOxford · 
United Kingdom104 works · 157,183 citations🔬Peng Cheng LaboratoryShenzhen · China38 works · 2,504 citationsFields, topics and concepts
📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works
🧭 Topics 2Research subjects and languages the work is about.Dialogue systems · 624 worksSpeech & spoken language · 480 works
Cite
Cite as Kuofeng Gao, Shu-Tao Xia, Ke Xu, Philip Torr and Jindong Gu. 2025. Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
@inproceedings{gao2025benchmarking,
title = {{Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models}},
author = {Kuofeng Gao, Shu-Tao Xia, Ke Xu, Philip Torr and Jindong Gu},
booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
year = {2025},
doi = {10.18653/v1/2025.acl-long.237},
url = {https://doi.org/10.18653/v1/2025.acl-long.237}
}References
- Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. \#\# Conclusion
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/gao-2025-benchmarking-open-ended-audio-dialogue/