Elamur Research⌕
☰
home / papers / 2026 / RSA-Bench: Benchmarking Audio Large Models in Real-World Acoustic Scenarios

RSA-Bench: Benchmarking Audio Large Models in Real-World Acoustic Scenarios

Paper · 2026 · Yibo Zhang, Kaiwen Luo, Liang Lin et al.

Summary

Audio Large Models (ALLMs) score well on clean benchmarks, but their behavior in realistic noisy settings has rarely been measured. Earlier robustness tests mostly relied on synthetic Gaussian noise or a single interfering source. RSA-Bench addresses this with more than 100,000 test items covering six tasks, from basic perception (ASR, gender and emotion recognition) to higher-level reasoning (spoken QA, math).

The authors build each item by mixing clean speech with one to four real environmental recordings. These come from the ESC subset of DynamicSuperb and are grouped into four scenes: Pasture, Extreme Weather, Classroom and Outdoors. Noise clips are looped or trimmed to match the speech length and scaled to the same RMS energy as the speech.

Across the evaluated models, accuracy falls as more noise sources are added. Reasoning tasks degrade far more sharply than perception tasks, and speech-like background sounds, such as children playing, cause more damage than mechanical noise. Applying standard speech enhancement before inference usually makes results worse. The authors argue that robustness must therefore be built into the models during training rather than added through preprocessing.

Key points

  • Benchmark design: RSA-Bench has 6 tasks × 4 scenes × 4 complexity levels (K = 1–4 noise sources), giving 16 variants per clean sample. The largest tasks are ASR and gender recognition (2,600 base samples each), and clean audio comes from LibriSpeech, IEMOCAP, MELD, SpokenMQA, SLUE Phase-2 and OpenHermes.
  • Perception–cognition gap: Low-level skills such as gender recognition stay relatively stable under noise, while reasoning tasks collapse. For example, SeaLLMs drops from 62.00 on clean audio to 10.00 at K = 3.
  • Denoising methods tested: noisereduce, RNNoise, Audio-Denoising (wavelet-based) and DeepFilterNet were applied to Qwen2-Audio, MERaLiON and StepAudio2 in the Pasture and Classroom scenes.
  • Denoising often hurts: At K = 1, Qwen2-Audio's ASR WER rises from 4.24% on noisy audio to 12.90% after noisereduce. At K = 4, noisereduce pushes WER to 55.74%, compared with 14.63% on the unprocessed noisy audio.
  • Deep learning vs. traditional enhancement: Neural enhancers such as DeepFilterNet do less damage than traditional signal-processing methods, but none of them beat the raw noisy input.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations🏛️Chinese Academy of SciencesBeijing · China246 works · 24,952 citations🎓Beijing University of Posts and TelecommunicationsBeijing · China78 works · 33,365 citations🔬Institute of Information EngineeringBeijing · China41 works · 1,485 citations🎓Hangzhou Dianzi UniversityHangzhou · China15 works · 1,061 citations🎓Shanghai UniversityShanghai · China6 works · 124 citations🎓North China Electric Power UniversityBeijing · China5 works · 1 citation

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 519 works

Cite

Cite as Yibo Zhang, Kaiwen Luo, Liang Lin, Shilinlu Yan, Jin Wang, Yaoqi Guo, Yitian Chen, Yalan Qin, Zhenhong Zhou, Kun Wang and Li Sun. 2026. RSA-Bench: Benchmarking Audio Large Models in Real-World Acoustic Scenarios. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.913

@inproceedings{zhang2026rsa,
  title = {{RSA-Bench: Benchmarking Audio Large Models in Real-World Acoustic Scenarios}},
  author = {Yibo Zhang, Kaiwen Luo, Liang Lin, Shilinlu Yan, Jin Wang, Yaoqi Guo, Yitian Chen, Yalan Qin, Zhenhong Zhou, Kun Wang and Li Sun},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.913},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.913}
}

References

57 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, and 1 others. 2025. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.…
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Orevaoghene Ahia, Martijn Bartelds, Kabir Ahuja, Hila Gonen, Valentin Hofmann, Siddhant Arora, Shuyue Stella Li, Vishal Puttagunta, Mofetoluwa Adeyemi, Charishma Buchireddy, and 1 others. 2025. Blab: Brutally long audio bench. arXiv preprint arXiv:2505.03054. — [arXiv:2505.03054](https://arxiv.org/abs/2505.03054) find
  4. Harsh Ahlawat, Naveen Aggarwal, and Deepti Gupta. 2025. Automatic speech recognition: A survey of deep learning techniques and approaches. International Journal of Cognitive Computing in Engineering. find
  5. Joseph Ozigis Akomodi, Maddisyn Neita, Samia Ahmed, Aiyana Parnell, Aaliyah Taylor, Aaliya Taylor, and Neita Maddisyn. 2025. Statistical data analysis of signal-to-noise ratios in mri and ct scans. International Journal of Recent Innovations in Academic Research, 9(3):240–250. find
  6. MA Ali and PM Shemi. 2015. An improved method of audio denoising based on wavelet transform. In 2015 international conference on Power, Instrumentation, Control and Computing (PICC), pages 1–6. IEEE. find
  7. Hanin Atwany, Abdul Waheed, Rita Singh, Monojit Choudhury, and Bhiksha Raj. 2025. Lost in transcription, found in distribution shift: Demystifying hallucination in speech foundation models. arXiv preprint arXiv:2502.12414. find
  8. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359. find
  9. Guangke Chen, Fu Song, Zhe Zhao, Xiaojun Jia, Yang Liu, Yanchen Qiao, and Weizhe Zhang. 2025. Audiojailbreak: Jailbreak attacks against end-to-end large audio-language models. arXiv preprint arXiv:2505.14103. — [arXiv:2505.14103](https://arxiv.org/abs/2505.14103) find
  10. Sujal Chondhekar, Vasanth Murukuri, Rushabh Vasani, Sanika Goyal, Rajshree Badami, Anushree Rana, Sanjana SN, Karthik Pandia, Sulabh Katiyar, Neha Jagadeesh, and 1 others. 2025. When de-noising hurts: A systematic study of speech enhancement effects on modern medical asr systems. arXiv preprint arXi… — [arXiv:2512.17562](https://arxiv.org/abs/2512.17562) find
  11. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, and 1 others. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.
  12. Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent. 2020. Librimix: An open-source dataset for generalizable speech separation. arXiv preprint arXiv:2005.11262. — [arXiv:2005.11262](https://arxiv.org/abs/2005.11262) find
  13. Mohsen Fatehifar, Josef Schlittenlacher, Ibrahim Almu-farrij, David Wong, Tim Cootes, and Kevin J Munro. 2025. Applications of automatic speech recognition and text-to-speech technologies for hearing assessment: a scoping review. International Journal of Audiology, 64(6):537–548. find
  14. Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and 1 others. 2025. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. arXiv preprint arXiv:2507.08128. — [arXiv:2507.08128](https://arxiv.org/abs/2507.08128) find
  15. Xinlu He and Jacob Whitehill. 2025. Survey of end-to-end multi-speaker automatic speech recognition for monaural audio. arXiv preprint arXiv:2505.10975. — [arXiv:2505.10975](https://arxiv.org/abs/2505.10975) find
  16. Yingxu He, Zhuohan Liu, Shuo Sun, Bin Wang, Wenyu Zhang, Xunlong Zou, Nancy F Chen, and Ai Ti Aw. 2024. Meralion-audiollm: Bridging audio and language with large language models. arXiv preprint arXiv:2412.09818. — [arXiv:2412.09818](https://arxiv.org/abs/2412.09818) find
  17. Gongping Huang, Jesper R Jensen, Jingdong Chen, Jacob Benesty, Mads G Christensen, Akihiko Sugiyama, Gary Elko, and Tomas Gaensler. 2025. Advances in microphone array processing and multi-channel speech enhancement. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Pr… find
  18. Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, and 1 others. 2024. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence, vo… find
  19. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  20. Chaitanya Jannu and Sunny Dayal Vanambathina. 2025. An overview of speech enhancement based on deep learning techniques. International Journal of Image and Graphics, 25(01):2550001. find
  21. Sonal Kumar, Šimon Sedláček, Vaibhavi Lokegaonkar, Fernando López, Wenyi Yu, Nishit Anand, Hyeonggon Ryu, Lichang Chen, Maxim Plička, Miroslav Hlaváček, and 1 others. 2025. Mmau-pro: A challenging and comprehensive benchmark for holistic evaluation of audio general intelligence. arXiv preprint ar… — [arXiv:2508.13992](https://arxiv.org/abs/2508.13992) find
  22. Chen-An Li, Tzu-Han Lin, and Hung-yi Lee. 2025a. When silence matters: The impact of irrelevant audio on text reasoning in large audio-language models. arXiv preprint arXiv:2510.00626. — [arXiv:2510.00626](https://arxiv.org/abs/2510.00626) find
  23. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, and 1 others. 2025b. From generation to judgment: Opportunities and challenges of llm-as-a-judge. In Proceedings of the 2025 Conference on Empirical Method… find
  24. Chaoqun Liu, Mahani Aljunied, Guizhen Chen, Hou Pong Chan, Weiwen Xu, Yu Rong, and Wenxuan Zhang. 2025a. Seallms-audio: Large audio-language models for southeast asia. arXiv preprint arXiv:2511.01670. — [arXiv:2511.01670](https://arxiv.org/abs/2511.01670) find
  25. Yao Liu, Faizahani binti Ab Rahman, and Farah binti Mohamad Zain. 2025b. A systematic literature review of research on automatic speech recognition in efl pronunciation. Cogent Education, 12(1):2466288. find
  26. Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, and 1 others. 2025. Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix. arXiv preprint arXiv:2505.13032. — [arXiv:2505.13032](https://arxiv.org/abs/2505.13032) find
  27. Matthew Maciejewski, Gordon Wichern, Emmett Mc-Quinn, and Jonathan Le Roux. 2020. Whamr!: Noisy and reverberant single-channel speech separation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 696–700. IEEE. find
  28. Daniela Silvia Pace, Giulia Pedrazzi, Irene D’amario, Aurora Troccoli, Giancarlo Giacomini, Maria Silvia Labriola, Gianni Pavan, Daniele Ventura, Edoardo Casoli, Giandomenico Ardizzone, and 1 others. 2025. The acoustic ecology of coastal dolphins by assessing the structural variability of sounds and… find
  29. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  30. Zifan Peng, Yule Liu, Zhen Sun, Mingchen Li, Zeren Luo, Jingyi Zheng, Wenhan Dong, Xinlei He, Xuechao Wang, Yingjie Xue, and 1 others. 2025. Jalmbench: Benchmarking jailbreak vulnerabilities in audio language models. arXiv preprint arXiv:2505.17568. — [arXiv:2505.17568](https://arxiv.org/abs/2505.17568) find
  31. Karol J Piczak. 2015. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015– 1018. find
  32. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 527–536.
  33. Tim Sainburg, Marvin Thielk, and Timothy Q Gentner. 2020. Finding, visualizing, and quantifying latent structure across diverse animal vocal repertoires. PLoS computational biology, 16(10):e1008228. find
  34. Mohammad Sarim, Saim Shakeel, Laeeba Javed, Mohammad Nadeem, and 1 others. 2025. Direct speech to speech translation: A review. arXiv preprint arXiv:2503.04799. — [arXiv:2503.04799](https://arxiv.org/abs/2503.04799) find
  35. Hendrik Schröter, Tobias Rosenkranz, Alberto N Escalante-B, and Andreas Maier. 2023. Deepfilternet: Perceptually motivated real-time speech enhancement. arXiv preprint arXiv:2305.08227. — [arXiv:2305.08227](https://arxiv.org/abs/2305.08227) find
  36. Suwon Shon, Siddhant Arora, Chyi-Jiunn Lin, Ankita Pasad, Felix Wu, Roshan Sharma, Wei-Lun Wu, Hung-yi Lee, Karen Livescu, and Shinji Watanabe. 2023. Slue phase-2: A benchmark suite of diverse spoken language understanding tasks. In Proceedings of the 61st Annual Meeting of the Association for Compu… find
  37. Jinhui Song, Xingxing Shi, Jiu Hui Wu, Tengyue Zheng, and Zhiwei Song. 2025. Nonlinear multi-order coupled stochastic resonance modeling under extremely low signal-to-noise ratios. Mechanical Systems and Signal Processing, 224:112208. find
  38. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289.
  39. Jean-Marc Valin. 2018. A hybrid dsp/deep learning approach to real-time full-band speech enhancement. In 2018 IEEE 20th international workshop on multimedia signal processing (MMSP), pages 1–5. IEEE. find
  40. Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy Chen. 2025a. Audiobench: A universal benchmark for audio large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin…
  41. Wei Wang, Siyi Zhao, and Yanmin Qian. 2025b. Advancing non-intrusive suppression on enhancement distortion for noise robust asr. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  42. Chengwei Wei, Bin Wang, Jung-jae Kim, and Nancy F Chen. 2025. Towards spoken mathematical reasoning: Benchmarking speech-based models over multi-faceted math problems. arXiv preprint arXiv:2505.15000. — [arXiv:2505.15000](https://arxiv.org/abs/2505.15000) find
  43. Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. 2019. Wham!: Extending speech separation to noisy environments. arXiv preprint arXiv:1907.01160. find
  44. Kendall Wrightson. 2000. An introduction to acoustic ecology. Soundscape: The journal of acoustic ecology, 1(1):10–13. find
  45. Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, and 1 others. 2025. Step-audio 2 technical report. arXiv preprint arXiv:2507.16632.
  46. Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. 2025. Audio-reasoner: Improving reasoning capability in large audio language models. arXiv preprint arXiv:2503.02318.
  47. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025b. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765.
  48. Chih-Kai Yang, Neo S Ho, and Hung-yi Lee. 2025a. Towards holistic evaluation of large audio-language models: A comprehensive survey. arXiv preprint arXiv:2505.15957.
  49. Hao Yang, Lizhen Qu, Ehsan Shareghi, and Gholamreza Haffari. 2025b. Audio is the achilles’ heel: Red teaming audio large multimodal models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volu… find
  50. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and 1 others. 2024. Air-bench: Benchmarking large audio-language models via generative comprehension. arXiv preprint arXiv:2402.07729.
  51. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, and 1 others. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800.
  52. Sally Taha Yousif and Basheera M Mahmmod. 2025. Speech enhancement algorithms: A systematic literature review. Algorithms, 18(5):272. find
  53. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000.
  54. Haomin Zhang, Chang Liu, Junjie Zheng, Zihao Chen, Chaofan Ding, and Xinhan Di. 2025. Deepaudio-v1: Towards multi-modal multi-stage end-to-end video to speech and audio generation. arXiv preprint arXiv:2503.22265. — [arXiv:2503.22265](https://arxiv.org/abs/2503.22265) find
  55. Yibo Zhang and Liang Lin. 2025. Enj: Optimizing noise with genetic algorithms to jailbreak lsms. arXiv preprint arXiv:2509.11128. — [arXiv:2509.11128](https://arxiv.org/abs/2509.11128) find
  56. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.
  57. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, and 1 others. 2025a. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. — Qwen2.5-Omni Technical Report

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-rsa-bench-benchmarking-audio-large/