Elamur Research⌕
☰
home / papers / 2026 / SEE: Signal Embedding Energy for Quantifying Noise Interference in Large Audio Language Models

SEE: Signal Embedding Energy for Quantifying Noise Interference in Large Audio Language Models

Paper · 2026 · Yuanhe Zhang, Jiayu Tian, Yibo Zhang et al.

Summary

Large Audio Language Models (LALMs) used in settings like car assistants or meeting tools often receive audio degraded by device and environmental noise. Until now, robustness has mostly been judged indirectly through large benchmark runs, which gives little insight into how noise affects the model internally. The authors propose Signal Embedding Energy (SEE), an internal metric. Offline, it compares mean-pooled activations from clean requests and pure-noise recordings, finds the layers where noise dominates, and uses SVD to extract noise-only directions that form a noise subspace. Online, SEE measures how much of an input's activation energy falls into that subspace. SEE rises steadily with noise level and tracks generation quality closely. The study also finds that standard speech denoisers barely help LALMs and can even increase SEE. Building on this, the training-free SEEN method removes the noise-subspace projection at embedding layers and beats existing denoisers. The work offers a model-aware way to diagnose and reduce noise interference without retraining.

Key points

  • SEE correlates strongly with LALM task performance, with a Pearson coefficient of 0.98.
  • Conventional denoisers (e.g., DFL, SEGAN, STFT- and wavelet-based methods) often fail to lower SEE, suggesting acoustic cleanup does not match the model's semantic sensitivity.
  • SEEN raises accuracy by 6.7% on average over traditional and learned denoisers, with negligible losses on clean input. It was tested on Qwen, MiniCPM and StepAudio models across MMAU speech, sound and music subsets.
  • Ablations show that restricting SEE to early or middle layers gives few valid directions and unstable detection. A neutralization strength λ above 1 introduces counter-noise distortions, and raising the cosine threshold δ (default 0.1) tends to harm the main semantic directions.
  • Limitations: the method requires pure-noise recordings from the deployment environment, and its mean pooling may miss temporal or paralinguistic corruption. Code is available at github.com/jyutian/SEEN.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore156 works · 25,644 citations🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🎓Beijing University of Posts and TelecommunicationsBeijing · China57 works · 32,843 citations🔬Institute of Information EngineeringBeijing · China30 works · 1,411 citations🎓North China Electric Power UniversityBeijing · China3 works · 1 citation🎓Chongqing University of Posts and TelecommunicationsChongqing · China2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSpeech & spoken language · 480 works

Cite

Cite as Yuanhe Zhang, Jiayu Tian, Yibo Zhang, Shilinlu Yan, Liang Lin, Zhenhong Zhou, Li Sun and Sen Su. 2026. SEE: Signal Embedding Energy for Quantifying Noise Interference in Large Audio Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.866

@inproceedings{zhang2026see,
  title = {{SEE: Signal Embedding Energy for Quantifying Noise Interference in Large Audio Language Models}},
  author = {Yuanhe Zhang, Jiayu Tian, Yibo Zhang, Shilinlu Yan, Liang Lin, Zhenhong Zhou, Li Sun and Sen Su},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.866},
  url = {https://doi.org/10.18653/v1/2026.acl-long.866}
}

References

49 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. In Noise reduction in speech processing, pages 1–4. Springer. find
  2. Steven Boll. 2003. Suppression of acoustic noise in speech using spectral subtraction. IEEE Transactions on acoustics, speech, and signal processing, 27(2):113–120. find
  3. Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. 2023. Beats: Audio pre-training with acoustic tokenizers. In International Conference on Machine Learning, pages 5178–5193. PMLR. find
  4. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, and 1 others. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.
  5. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919.
  6. Dimitar D Deliyski, Heather S Shaw, and Maegan K Evans. 2005. Adverse effects of environmental noise on acoustic voice quality measurements. Journal of Voice, 19(1):15–28. find
  7. David L Donoho. 2002. De-noising by soft-thresholding. IEEE transactions on information theory, 41(3):613–627. find
  8. Yariv Ephraim and David Malah. 2003. Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator. IEEE Transactions on acoustics, speech, and signal processing, 32(6):1109–1121. find
  9. David Gelbart and Nelson Morgan. 2002. Double the trouble: handling noise and reverberation in far-field automatic speech recognition. In Interspeech, pages 2185–2188. find
  10. Karan Goel, Nazneen Rajani, Jesse Vig, Zachary Taschdjian, Mohit Bansal, and Christopher Ré. 2021. Robustness gym: Unifying the nlp evaluation landscape. find
  11. Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. 2024. Listen, think, and understand. In International Conference on Learning Representations. find
  12. Shreyas Gopal, Ashutosh Anshul, Haoyang Li, Yue Heng Yeo, Hexin Liu, and Eng Siong Chng. 2025. Explainable disentanglement on discrete speech representations for noise-robust asr. arXiv preprint arXiv:2510.25150. — [arXiv:2510.25150](https://arxiv.org/abs/2510.25150) find
  13. Guanyu Hou, Jiaming He, Yinhang Zhou, Ji Guo, Yitong Qiao, Rui Zhang, and Wenbo Jiang. 2025. Evaluating robustness of large audio language models to audio injection: An empirical study. arXiv preprint arXiv:2505.19598. find
  14. Guoning Hu and DeLiang Wang. 2010. A tandem algorithm for pitch estimation and voiced speech segregation. IEEE Transactions on Audio, Speech, and Language Processing, 18(8):2067–2079. find
  15. Yuchen Hu, Chen Chen, Chao-han Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen, and Ensiong Chng. 2024. Large language models are efficient learners of noise-robust speech recognition. In International Conference on Learning Representations. find
  16. Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. 2015. Audio augmentation for speech recognition. In Proc. find
  17. Interspeech 2015, pages 3586–3589. find
  18. Pankaj Kumar and Subhankar Mishra. 2025. Robustness in large language models: A survey of mitigation strategies and evaluation metrics. arXiv preprint arXiv:2505.18658. — [arXiv:2505.18658](https://arxiv.org/abs/2505.18658) find
  19. Sonal Kumar, Šimon Sedláček, Vaibhavi Lokegaonkar, Fernando López, Wenyi Yu, Nishit Anand, Hyeong-gon Ryu, Lichang Chen, Maxim Plička, Miroslav Hlaváček, and 1 others. 2025. Mmau-pro: A challenging and comprehensive benchmark for holistic evaluation of audio general intelligence. arXiv preprint a… — [arXiv:2508.13992](https://arxiv.org/abs/2508.13992) find
  20. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and 1 others. 2021. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336–1…
  21. Bohan Li, Wenbin Huang, Yuhang Qiu, Yiwei Guo, Hankun Wang, Zhihan Li, Jing Peng, Ziyang Ma, Xie Chen, and Kai Yu. 2025a. Isa-bench: Benchmarking instruction sensitivity for large audio language models. arXiv preprint arXiv:2510.23558. — [arXiv:2510.23558](https://arxiv.org/abs/2510.23558) find
  22. Chen-An Li, Tzu-Han Lin, and Hung-yi Lee. 2025b. When silence matters: The impact of irrelevant audio on text reasoning in large audio-language models. arXiv preprint arXiv:2510.00626. — [arXiv:2510.00626](https://arxiv.org/abs/2510.00626) find
  23. Jing Li, Jingyuan Li, Guo Yang, Lie Yang, Haozhuang Chi, and Lichao Yang. 2025c. Applications of large language models and multimodal large models in autonomous driving: A comprehensive review. Drones, 9(4):238. find
  24. Jinyu Li, Li Deng, Reinhold Haeb-Umbach, and Yifan Gong. 2015. Robust automatic speech recognition: a bridge to practical applications. Academic Press. find
  25. Jae Lim. 1978. All-pole modeling of degraded speech. IEEE Transactions on Acoustics, Speech, and Signal Processing, 26(3):197–210. find
  26. Liang Lin, Miao Yu, Kaiwen Luo, Yibo Zhang, Lilan Peng, Dexian Wang, Xuehai Tang, Yuanhe Zhang, Xikang Yang, Zhenhong Zhou, and 1 others. 2025. Hidden in the noise: Unveiling backdoors in audio llms alignment through latent acoustic pattern triggers. arXiv preprint arXiv:2508.02175. — [arXiv:2508.02175](https://arxiv.org/abs/2508.02175) find
  27. Yen-Ju Lu, Zhong-Qiu Wang, Shinji Watanabe, Alexander Richard, Cheng Yu, and Yu Tsao. 2022. Conditional diffusion probabilistic model for speech enhancement. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7402–7406. Ieee. find
  28. Yi Luo and Nima Mesgarani. 2019. Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation. IEEE/ACM transactions on audio, speech, and language processing, 27(8):1256–1266. find
  29. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pages 5206–5210. IEEE. find
  30. Pavlos Papadopoulos, Andreas Tsiartas, and Shrikanth Narayanan. 2016. Long-term snr estimation of speech signals in known and unknown channel conditions. IEEE/ACM Transactions on audio, speech, and language processing, 24(12):2495–2506. find
  31. Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. In Interspeech, pages 2613–2617.
  32. Santiago Pascual, Antonio Bonafonte, and Joan Serrà. 2017. Segan: Speech enhancement generative adversarial network. In Proc. find
  33. Interspeech 2017, pages 3642–3646. find
  34. Jing Peng, Yucheng Wang, Yangui Fang, Yu Xi, Xu Li, Xizhuo Zhang, and Kai Yu. 2024. A survey on speech large language models. arXiv preprint arXiv:2410.18908. — [arXiv:2410.18908](https://arxiv.org/abs/2410.18908) find
  35. Hendrik Purwins, Bo Li, Tuomas Virtanen, Jan Schlüter, Shuo-Yiin Chang, and Tara Sainath. 2019. Deep learning for audio signal processing. IEEE Journal of Selected Topics in Signal Processing, 13(2):206– 219. find
  36. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  37. Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, and 1 others. 2023. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925.
  38. Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. 2024. Taskbench: Benchmarking large language models for task automation. Advances in Neural Information Processing Systems, 37:4540–4574. find
  39. Gilbert W Stewart. 1993. On the early history of the singular value decomposition. SIAM review, 35(4):551– 566. find
  40. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289.
  41. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, and 1 others. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800.
  42. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15757–15773.
  43. Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy Chen. 2025. Audiobench: A universal benchmark for audio large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Ling…
  44. DeLiang Wang and Jitong Chen. 2018. Supervised speech separation based on deep learning: An overview. IEEE/ACM transactions on audio, speech, and language processing, 26(10):1702–1726. find
  45. Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, and 1 others. 2018. Espnet: End-to-end speech processing toolkit. In Proceedings of the Annual Conference of the International Speech Com… find
  46. Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, Mingrui Chen, Peng Liu, Wang You, Xiangyu Tony Zhang, Xingyuan Li, Xuerui Yang, Yayue Deng, Yechang Huang, Yuxin Li, and 90 others. 2025. Step-audio 2 technical report. Preprint, arXiv:250…
  47. Zhen Xiong, Yujun Cai, Zhecheng Li, Junsong Yuan, and Yiwei Wang. 2025. Thinking with sound: Audio chain-of-thought enables multimodal reasoning in large audio-language models. arXiv preprint arXiv:2509.21749.
  48. Yong Xu, Jun Du, Li-Rong Dai, and Chin-Hui Lee. 2014. A regression approach to speech enhancement based on deep neural networks. IEEE/ACM transactions on audio, speech, and language processing, 23(1):7–19. find
  49. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and 1 others. 2024. Air-bench: Benchmarking large audio-language models via generative comprehension. In Proceedings of the 62nd Annual Meeting of the Association for Computational…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-see-signal-embedding-energy-quantifying/