Elamur Research⌕
☰
home / papers / 2026 / Discovering and Causally Validating Emotion-Sensitive Neurons in Large Audio-Language Models

Discovering and Causally Validating Emotion-Sensitive Neurons in Large Audio-Language Models

Paper · 2026 · Xiutian Zhao, Björn Schuller and Berrak Sisman

Summary

The paper asks whether large audio-language models (LALMs) store emotion information in small, identifiable groups of neurons, or whether it is spread diffusely across the network. The authors study three open-source models: Qwen2.5-Omni-7B, Kimi-Audio and Audio Flamingo 3. They use speech emotion recognition (SER) on IEMOCAP, MELD and MSP-PODCAST as a diagnostic task.

The method has three steps. First, they record SwiGLU gate activations in the decoder MLPs, using only items the model answers correctly. Second, they score neurons with several selectors: frequency (LAP), entropy (LAPE), mean deviation (MAD) and contrast (ConAct), compared against a random baseline. Third, they intervene at inference time, either by switching the selected neurons off or by amplifying them.

Switching off the neurons tied to one emotion mainly hurts recognition of that emotion and leaves other classes largely intact. Amplifying the same neurons pushes predictions toward that emotion. MAD and ConAct isolate these units far better than LAP or LAPE. The authors present this as the first causal, neuron-level account of emotion in LALMs. It suggests a practical way to control affective behaviour in speech assistants.

Key points

  • Self–cross effect: For emotion-sensitive neurons (ESNs) chosen by MAD or ConAct, deactivation causes a strong drop for the target emotion ("self") and much smaller drops for other emotions ("cross"). The effects grow with intervention strength and need only modest identification data.
  • Weaker selectors and baseline: LAP and LAPE often fail to isolate causally specific units. The random baseline (RND) samples the same neuron budget and is averaged over 5 seeds.
  • Steering: Targeted amplification reliably shifts predictions toward the chosen emotion. Label-free injection was also tested in three variants (2-Pass, Mix, Union) and gave mixed results, which hints that ESNs compete with each other when boosted jointly.
  • Layer locality: ESNs cluster in decoder layer 0, layers 6–8 and layers 19–22. They are sparse in the central layers 15–18.
  • Cross-dataset transfer: ESNs identified on one corpus partly transfer to another, but the effect is uneven and sometimes asymmetric. Neutral transfers least stably, so the authors recommend identifying ESNs on multiple datasets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🎓Imperial College LondonLondon · United Kingdom48 works · 12,181 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSentiment & emotion · 972 worksSpeech & spoken language · 480 worksInterpretability · 130 works

Cite

Cite as Xiutian Zhao, Björn Schuller and Berrak Sisman. 2026. Discovering and Causally Validating Emotion-Sensitive Neurons in Large Audio-Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.687

@inproceedings{zhao2026discovering,
  title = {{Discovering and Causally Validating Emotion-Sensitive Neurons in Large Audio-Language Models}},
  author = {Xiutian Zhao, Björn Schuller and Berrak Sisman},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.687},
  url = {https://doi.org/10.18653/v1/2026.acl-long.687}
}

References

53 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mehmet Berkehan Akçay and Kaya Oğuz. 2020. Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers. Speech Communication, 116:56–76. find
  2. Alican Akman, Qiyang Sun, and Björn W. Schuller. 2025. Improving audio explanations using audio language models. IEEE Signal Processing Letters, 32:741–745. find
  3. Badr AlKhamissi, Greta Tuckute, Antoine Bosselut, and Martin Schrimpf. 2025. The LLM language network: A neuroscientific approach for identifying causally task-relevant units. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistic… find
  4. Omer Antverg and Yonatan Belinkov. 2022. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations. find
  5. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations. find
  6. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Network Dissection: Quantifying Interpretability of Deep Visual Representations . In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3319–3327, Los Alamitos, CA, USA. IEEE Computer Society. find
  7. David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. 2020. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 117(48):30071–30078. find
  8. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335–359. find
  9. Carlos Busso, Reza Lotfian, Kusha Sridhar, Ali N. Salman, Wei-Cheng Lin, Lucas Goncalves, Srinivas Parthasarathy, Abinay Reddy Naini, Seong-Gyun Leem, Luz Martinez-Lucas, Huang-Cheng Chou, and Pravin Mote. 2026. The msp-podcast corpus. IEEE Transactions on Affective Computing, pages 1–19. find
  10. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. 2019. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applica… find
  11. Junfeng Fang, Zongze Bi, Ruipeng Wang, Houcheng Jiang, Yuan Gao, Kun Wang, An Zhang, Jie Shi, Xiang Wang, and Tat-Seng Chua. 2024. Towards neuron attributions in multimodal large language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24,… find
  12. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. find
  13. Kanishk Gandhi, J.-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. 2023. Understanding social reasoning in language models with language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. find
  14. Sreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. 2025. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. In The Thirty-ninth Annual Con… find
  15. Yves Grandvalet and Yoshua Bengio. 2004. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems, volume 17. MIT Press. find
  16. Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. 2024. Universal neurons in GPT2 language models. Transactions on Machine Learning Research. find
  17. Kaichen Huang, Jiahao Huo, Yibo Yan, Kun Wang, Yutao Yue, and Xuming Hu. 2024. Miner: Mining the underlying pattern of modality-specific neurons in multimodal large language models. Preprint, arXiv:2410.04819. — [arXiv:2410.04819](https://arxiv.org/abs/2410.04819) find
  18. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations. find
  19. Jiahao Huo, Yibo Yan, Boren Hu, Yutao Yue, and Xuming Hu. 2024. MMNeuron: Discovering neuron-level domain-specific interpretation in multimodal large language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6801–6816, Miami, Florida, USA. Assoc… find
  20. KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, and 21 others. 2025. Kimi-audio technical report. Preprint, arXiv:2504.18425.
  21. Jaewook Lee, Woojin Lee, Oh-Woog Kwon, and Harksoo Kim. 2025. Do large language models have “emotion neurons”? investigating the existence and role. In Findings of the Association for Computational Linguistics: ACL 2025, pages 15617–15639, Vienna, Austria. Association for Computational Linguistics. find
  22. Alexander H. Liu, Andy Ehrenberg, Andy Lo, Clément Denoix, Corentin Barreau, Guillaume Lample, Jean-Malo Delignon, Khyathi Raghavi Chandu, Patrick von Platen, Pavankumar Reddy Muddireddy, Sanchit Gandhi, Soham Ghosh, Srijan Mishra, Thomas Foubert, Abhinav Rastogi, Adam Yang, Albert Q. Jiang, Alexand… find
  23. Rui Liu, Berrak Sisman, Guanglai Gao, and Haizhou Li. 2021. Expressive tts training with frame and style reconstruction loss. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1806–1818. find
  24. Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, ShiLiang Zhang, and Xie Chen. 2024. emotion2vec: Self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15747–15760, Bangkok, Thailand. Association for… find
  25. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152–159, New York City, USA. Association for Computational Linguistics.
  26. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc. find
  27. Abinay Reddy Naini, Lucas Goncalves, Ali N. Salman, Pravin Mote, Ismail R. Ulgen, Thomas Thebaud, Laureano Moro Velazquez, Leibny Paola Garcia, Najim Dehak, Berrak Sisman, and Carlos Busso. 2025. find
  28. The Interspeech 2025 Challenge on Speech Emotion Recognition in Naturalistic Conditions. In Interspeech 2025, pages 4668–4672. find
  29. Danial Namazifard and Lukas Galke Poech. 2025. Isolating culture neurons in multilingual large language models. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics… find
  30. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527– 536, Flore…
  31. Noam Shazeer. 2020. Glu variants improve transformer. Preprint, arXiv:2002.05202. find
  32. Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations. find
  33. Yaman Kumar Singla, Jui Shah, Changyou Chen, and Rajiv Ratn Shah. 2022. What do audio transformers hear? probing their representations for language delivery & structure. In 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pages 910–925. find
  34. Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu…
  35. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2024. Steering language models with activation engineering. Preprint, arXiv:2308.10248.
  36. Ismail Rasim Ulgen, Zongyang Du, Carlos Busso, and Berrak Sisman. 2024. Revealing emotional clusters in speaker embeddings: A contrastive learning strategy for speech emotion recognition. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1… find
  37. Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. 2024. Neurons in large language models: Dead, n-gram, positional. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1288–1301, Bangkok, Thailand. Association for Computational Linguistics. find
  38. Chenxi Wang, Yixuan Zhang, Ruiji Yu, Yufei Zheng, Lang Gao, Zirui Song, Zixiang Xu, Gus Xia, Huishuai Zhang, Dongyan Zhao, and Xiuying Chen. 2025. Do llms "feel"? emotion circuits discovery and control. Preprint, arXiv:2510.11328. — [arXiv:2510.11328](https://arxiv.org/abs/2510.11328) find
  39. Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. 2023a. Large-scale multi-modal pre-trained models: A comprehensive survey. Machine Intelligence Research, 20(4):447– 482. find
  40. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. Preprint, arXiv:2203.11171.
  41. Taiba Majid Wani, Teddy Surya Gunawan, Syed Asif Ahmad Qadri, Mira Kartiwi, and Eliathamby Ambikairajah. 2021. A comprehensive review of speech emotion recognition systems. IEEE access, 9:47795–47814. find
  42. Tung-Yu Wu, Yu-Xiang Lin, and Tsui-Wei Weng. 2024. And: audio network dissection for interpreting deep acoustic models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. find
  43. Jiaqi Xu, Cuiling Lan, and Yan Lu. 2025a. Deciphering functions of neurons in vision-language models. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 3173–3181. find
  44. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025b. Qwen2.5-omni technical report. Preprint, arXiv:2503.20215.
  45. Chih-Kai Yang, Neo Ho, Yi-Jyun Lee, and Hung-Yi Lee. 2025. Audiolens: A closer look at auditory attribute perception of large audio-language models. In 2025 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. find
  46. David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, pages 189–196, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
  47. Zeping Yu and Sophia Ananiadou. 2024. Neuron-level knowledge attribution in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3267–3280, Miami, Florida, USA. Association for Computational Linguistics. find
  48. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, volume 35, pages 15476–15488. Curran Associates, Inc.
  49. Xiutian Zhao, Rochelle Choenni, Rohit Saxena, and Ivan Titov. 2026a. Finding culture-sensitive neurons in vision-language models. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3366–3381, Rabat, Morocco. A…
  50. Xiutian Zhao, Ismail Rasim Ulgen, Philipp Koehn, Björn Schuller, and Berrak Sisman. 2026b. Neuron-level emotion control in speech-generative large audio-language models. Preprint, arXiv:2603.17231. — [arXiv:2603.17231](https://arxiv.org/abs/2603.17231) find
  51. Xiutian Zhao, Ke Wang, and Wei Peng. 2024. Measuring the inconsistency of large language models in preferential ranking. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 171–176, Bangkok, Thailand. Association for Computational Linguistics. find
  52. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations. find
  53. Kun Zhou, Berrak Sisman, Rajib Rana, Björn W. Schuller, and Haizhou Li. 2023. Speech synthesis with mixed emotions. IEEE Transactions on Affective Computing, 14(4):3120–3134. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-discovering-causally-validating-emotion-sensitive/