Elamur Research⌕
☰
home / papers / 2026 / Protecting Bystander Privacy via Selective Hearing in Audio LLMs

Protecting Bystander Privacy via Selective Hearing in Audio LLMs

Paper · 2026 · Xiao Zhan, Guangzhi Sun, Jose Such et al.

Summary

Audio LLMs running in voice assistants or wearables also pick up speech from nearby people who never chose to interact with them. Earlier privacy work protected only active users. This paper treats such bystanders as a distinct privacy risk and frames the desired behaviour as selective hearing: the model should follow the intended main speaker and decline any request about incidental background voices.

The authors build SH-Bench, which combines two kinds of audio. The first is real recordings made by paired Prolific participants reading scripts written by GPT-4o for five everyday settings. The second is synthetic mixtures built from the AMI Meeting Corpus. The audio comes with multiple-choice questions, and each question includes an "I don't know" option. Models are tested in a general mode and a selective mode. A new metric, Selective Efficacy (SE), scores comprehension and bystander protection together.

Current open-source and proprietary models leak a lot of bystander information, even when their audio understanding is strong. The proposed Bystander Privacy Fine-Tuning (BPFT) teaches models to refuse bystander questions while still understanding the main speaker. It clearly outperforms Gemini 2.5 Pro, the strongest model without fine-tuning.

Key points

  • SH-Bench size: 3,968 multi-speaker audio files, about 157.5 hours from 285 speakers.
    • Each test clip has 10 MCQs and each training clip has 20, giving 77.36k questions in total.
  • Real-audio partition: recorded by 100 Prolific participants in five scenarios: coffee shop, gym, shared living area, public transit and waiting room.
  • Synthetic partition: built from AMI individual headset microphone (IHM) recordings.
  • Main result: BPFT improves bystander accuracy in selective mode by 47 points and SE by 15.9 points over Gemini 2.5 Pro (SE 91.7% vs 75.8%).
  • Open-ended questions: after BPFT, refusal rates on bystander questions rise to 92% for Qwen-2.5-Omni 7B and 96% for Step-Audio-2-mini, up from 45% and 48%. Main-speaker questions are still almost never refused.
  • Limitations: the setup covers only a single main speaker, and the dataset will be released under a research-only, non-commercial licence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksSpeech & spoken language · 519 works

Cite

Cite as Xiao Zhan, Guangzhi Sun, Jose Such and Phil Woodland. 2026. Protecting Bystander Privacy via Selective Hearing in Audio LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.693

@inproceedings{zhan2026protecting,
  title = {{Protecting Bystander Privacy via Selective Hearing in Audio LLMs}},
  author = {Xiao Zhan, Guangzhi Sun, Jose Such and Phil Woodland},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.693},
  url = {https://doi.org/10.18653/v1/2026.acl-long.693}
}

References

55 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Forsad Al Hossain, M Tanjid Hasan Tonmoy, Andrew Lover, George Corey, Mohammad Arif Ul Alam, and Tauhidur Rahman. 2024. Crowdotic: A privacy-preserving hospital waiting room crowd density estimation with non-speech audio. In Proceedings of the 25th International Workshop on Mobile Computing Systems… find
  2. Antonios Alexos, Raghuveer Peri, Sai Muralidhar Jayanthi, Metehan Cekic, Srikanth Vishnubhotla, Kyu J Han, and Srikanth Ronanki. 2025. Defending speech-enabled LLMs against adversarial jailbreak threats. In Proc. Interspeech, pages 2048–2052. find
  3. Ranya Aloufi, Hamed Haddadi, and David Boyle. 2021. Configurable privacy-preserving automatic speech recognition. arXiv preprint arXiv:2104.00766. 15188 9 — [arXiv:2104.00766](https://arxiv.org/abs/2104.00766) find
  4. Ahmed Alshehri, Joseph Spielman, Amiya Prasad, and Chuan Yue. 2022. Exploring the privacy concerns of bystanders in smart homes from the perspectives of both owners and bystanders. Proceedings on Privacy Enhancing Technologies. find
  5. Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. 2023. SoK: Memorization in general-purpose large language models. arXiv preprint arXiv:2310.18362. — [arXiv:2310.18362](https://arxiv.org/abs/2310.18362) find
  6. Tom Bäckström. 2025. Privacy in speech technology. Proceedings of the IEEE. find
  7. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In Proc. ICLR.
  8. Lewis Birch. 2025. Audio-based jailbreak attacks on multi-modal llms. [Onine] Accessed: 21/10/2025. find
  9. Boyong Wu et al. 2025. Step-audio 2 technical report. arXiv:2507.16632.
  10. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650…
  11. Guangke Chen, Yedi Zhang, and Fu Song. 2024. SLMIA-SR: Speaker-level membership inference attacks against speaker recognition systems. In Proceedings of the 31st Annual Network and Distributed System Security (NDSS) Symposium. find
  12. Jiali Cheng and Hadi Amiri. 2025. Speech unlearning. In Proc. Interspeech. find
  13. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919.
  14. Comanici Gheorghe et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning. arXiv:2507.06261.
  15. Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent. 2020. Librimix: An open-source dataset for generalizable speech separation. arXiv preprint arXiv:2005.11262. — [arXiv:2005.11262](https://arxiv.org/abs/2005.11262) find
  16. Qingkai Fang, Yan Zhou, Shoutao Guo, Shaolei Zhang, and Yang Feng. 2025. Llama-omni 2: LLM-based real-time spoken chatbot with autoregressive streaming speech synthesis. In Proc. ACL.
  17. Daniel Garcia-Romero, David Snyder, Shinji Watanabe, Gregory Sell, Alan McCree, Daniel Povey, and Sanjeev Khudanpur. 2019. Speaker recognition benchmark using the CHiME-5 corpus. In Proc. Interspeech. find
  18. John J Godfrey, Edward C Holliman, and Jane Mc-Daniel. 1992. SWITCHBOARD: Telephone speech corpus for research and development. In Proc. ICASSP. find
  19. Pascal Hartig. 2025. Building multimodal AI for ray-ban meta glasses. [Onine] Accessed: 21/10/2025. find
  20. Huang, Chien-yu et al. 2024. Dynamic-superb: Towards a dynamic, collaborative, and comprehensive instruction-tuning benchmark for speech. In Proc. ICASSP. find
  21. Hurst, Aaron et al. 2024. GPT-4o system card. arXiv preprint arXiv:2410.21276.
  22. KimiTeam. 2025. Kimi-audio technical report. arXiv:2504.18425.
  23. Alkis Koudounas, Claudio Savelli, Flavio Giobergia, and Elena Baralis. 2025. “alexa, can you forget me?” machine unlearning benchmark in spoken language understanding. In Proc. Interspeech. find
  24. Wessel Kraaij, Thomas Hain, Mike Lincoln, and Wilfried Post. 2005. The AMI meeting corpus. In Proc. International Conference on Methods and Techniques in Behavioral Research, pages 1–4. find
  25. I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, W. Post, Dennis Reidsma, and P. Wellner. 2005. The AMI meeting corpus. In Proceedings of Measuring Behavior 2005, 5th International… find
  26. R Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. 2023. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. Transactions of the Association for Computational Linguistics, 11:652–670.
  27. Xiaoxiao Miao, Ruijie Tao, Chang Zeng, and Xin Wang. 2025. A benchmark for multi-speaker anonymization. IEEE Transactions on Information Forensics and Security. find
  28. Andreas Nautsch, Catherine Jasserand, Els Kindt, Massimiliano Todisco, Isabel Trancoso, and Nicholas Evans. 2019. The GDPR & speech data: Reflections of legal and technology communities, first steps towards a common understanding. In Proc. Interspeech. find
  29. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proc. ICML. 15189 10
  30. S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2024. MMAU: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410.19168.
  31. Eimaan Saqib, Shijing He, Junghyun Choy, Ruba Abu-Salma, Jose Such, Julia Bernd, and Mobin Javed. 2025. Bystander privacy in smart homes: A systematic review of concerns and solutions. ACM Trans. Comput.-Hum. Interact. find
  32. Anna Ståhlbröst, Annika Sällström, and Danilo Hollosi. 2014. Audio monitoring in smart cities: an information privacy perspective. IADIS International Association for Development of the Information Society. find
  33. Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong. 2021. “Attention Is All You Need” in Speech Separation: the SepFormer. In Proc. ICASSP. find
  34. Angela Sun. 2025. Gemini live: A more helpful, natural and visual assistant. [Onine] Accessed: 21/10/2025. find
  35. Yulin Sun, Qisheng Xu, Yi Su, Qian Zhu, Yong Dou, Xinwang Liu, and Kele Xu. 2025. Audioset-r: A refined audioset with multi-stage llm label reannotation. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 13089–13096. find
  36. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. Salmonn: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations.
  37. Larry W Thomas. 2018. Legal Implications of Video Surveillance on Transit Systems. TCRP Project J-05, Topic 17-02. Transportation Research Board. find
  38. Minh Tran and Mohammad Soleymani. 2023. Privacy-preserving representation learning for speech understanding. In Proc. Interspeech. find
  39. Wei-Cheng Tseng, Wei-Tsung Kao, and Hung-yi Lee. 2022. Membership inference attacks against self-supervised speech models. In Proc. Interspeech. find
  40. Tseng, Yuan et al. 2024. Av-superb: A multi-task evaluation benchmark for audio-visual representation models. In Proc. ICASSP. find
  41. Chen Wang, Minpeng Liao, Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. 2023. BLSP: Bootstrapping language-speech pre-training via behavior alignment of continuation writing. arXiv preprint arXiv:2309.00916.
  42. Lixu Wang, Kaixiang Yao, Xinfeng Li, Dong Yang, Haoyang Li, Xiaofeng Wang, and Wei Dong. 2025a. The man behind the sound: Demystifying audio private attribute profiling via multimodal large language model agents. arXiv preprint arXiv:2507.10016. — [arXiv:2507.10016](https://arxiv.org/abs/2507.10016) find
  43. Shuai Wang, Zhaokai Sun, Zhennan Lin, Chengyou Wang, Zhou Pan, and Lei Xie. 2025b. MSU-Bench: Towards understanding the conversational multi-talker scenarios. arXiv preprint arXiv:2508.08155. — [arXiv:2508.08155](https://arxiv.org/abs/2508.08155) find
  44. Wu, Jian et al. 2023. On decoder-only architecture for speech-to-text and large language model integration. In Proc. ASRU. find
  45. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen 2.5-omni technical report. arXiv:2503.20215.
  46. Wanqi Yang, Yanda Li, Meng Fang, Yunchao Wei, and Ling Chen. 2025a. Who can withstand chat-audio attacks? an evaluation benchmark for large audio-language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 17205– 17220. find
  47. Yudong Yang, Xuezhen Zhang, Zhifeng Han, Siyin Wang, Jimin Zhuang, Zengrui Jin, Jing Shao, Guangzhi Sun, and Chao Zhang. 2025b. Speech-audio compositional attacks on multimodal LLMs and their mitigation with SALMONN-Guard. arXiv preprint arXiv:2511.10222. — [arXiv:2511.10222](https://arxiv.org/abs/2511.10222) find
  48. Yudong Yang, Jimin Zhuang, Guangzhi Sun, Changli Tang, Yixuan Li, Peihan Li, Yifan Jiang, Wei Li, Zejun Ma, and Chao Zhang. 2025c. Audio-centric video understanding benchmark without text shortcut. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 6580–…
  49. Yaxing Yao, Justin Reed Basdeo, Oriana Rosata Mcdonough, and Yang Wang. 2019. Privacy perceptions and designs of bystanders in smart homes. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–24. find
  50. Yue, Xiang et al. 2024. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567.
  51. Jimena Zapata, Justin Sulik, Clemens von Wulffen, and Ophelia Deroy. 2024. Bystanders’ collective responses set the norm against hate speech. Humanities and Social Sciences Communications, 11(1):1–13. find
  52. Hossein Zeinali, Hossein Sameti, Themos Stafylakis, L Burget, and J Cernocky. 2018. Deepmine speech processing database: Text-dependent and. find
  53. Proc. Odyssey 2018 The Speaker and Language Recognition, pages 386–392. find
  54. Xiao Zhan, William Seymour, and Jose Such. 2024. Beyond individual concerns: multi-user privacy in large language models. In Proceedings of the 6th ACM Conference on Conversational User Interfaces, pages 1–6. 15190 11 find
  55. Main Speaker %Acc (Real/Synth) General Selective Bystander %Acc (Real/Synth) General Selective Gemini 2.5 Pro Qwen-2.5-Omni 7B Step-Audio-2-mini 97.6 / 97.0 95.6 / 96.4 94.8 / 93.6 97.4 / 96.6 95.6 / 95.4 94.0 / 93.4 54.4 / 76.6 41.4 / 55.0 51.0 / 58.6 68.2 / 50.2 53.0 / 42.2 35.8 / 27.2 Qwen-2.5-Om… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhan-2026-protecting-bystander-privacy-selective-hearing/