Elamur Research⌕
☰
home / papers / 2026 / Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations

Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations

Paper · 2026 · Sheng-Lun Wei, Yu-Ling Liao, Yen-Hua Chang et al.

Summary

Audio-capable multimodal LLMs now answer spoken questions directly. It was unclear whether their answers change depending on the speaker's gender, the language or accent, or the order of the answer options. The authors built BiasInEar, a spoken multiple-choice benchmark derived from Global MMLU Lite in English, Chinese and Korean.

To keep STEM content instead of discarding it, GPT OSS 120B first rewrote each item into a form that can be read aloud, covering formulas, chemical notation and fill-in blanks. Gemini 2.5 Flash Preview TTS then voiced the items with male and female voices and several regional accents. Quality was checked with two ASR systems and with human ratings.

Nine models from four families were scored with accuracy, entropy, APES and Fleiss' κ. The models stayed fairly stable across speaker gender but were sensitive to language and especially to option order. Reasoning strategy and architecture also affected robustness. Comparing text and audio inputs showed that speech mainly enlarges biases already present in text. The work gives a common framework for auditing fairness in speech-integrated LLMs.

Key points

  • Dataset scale: BiasInEar has 11,200 questions and 70.8 hours (≈4,249 minutes) of speech. Each question can appear in up to 28 configurations, combining:
    • language;
    • accent: American, British or Indian English; Beijing or Northeastern Mandarin; Seoul or Jeolla Korean;
    • gender: Gemini voices Orus and Zephyr;
    • option order: original or reversed.
  • Automatic quality checks: each clip was transcribed with Whisper Large v3 and Omnilingual ASR, keeping the lower WER of the two. Clips with zero WER made up 90.68% for English, 67.58% for Chinese and 68.79% for Korean.
  • Human quality checks: on sampled clips with nonzero WER, raters judged 82.65% (English), 86% (Chinese) and 75% (Korean) as fully correct.
  • Text vs. audio: audio input raised APES for every tested model. For Gemini 2.5 Flash Lite, APES rose from 0.081 to 0.178 under language variation and from 0.096 to 0.221 under option-order variation.
  • Pipeline vs. end-to-end: a transcribe-then-answer pipeline with Gemini 2.5 Flash and Gemma 3n E4B gave higher Fleiss' κ and lower APES than end-to-end use. This suggests that explicit transcription removes accent and gender cues.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Taiwan UniversityTaipei · Taiwan29 works · 2,116 citations🔬Academia SinicaTaipei · Taiwan10 works · 760 citations🔬Institute of Information Science, Academia SinicaTaipei · Taiwan7 works · 51 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSpeech & spoken language · 480 works

Cite

Cite as Sheng-Lun Wei, Yu-Ling Liao, Yen-Hua Chang, Hen-Hsen Huang and Hsin-Hsi Chen. 2026. Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.80

@inproceedings{wei2026bias,
  title = {{Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations}},
  author = {Sheng-Lun Wei, Yu-Ling Liao, Yen-Hua Chang, Hen-Hsen Huang and Hsin-Hsi Chen},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.80},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.80}
}

References

44 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude 3.7 sonnet and claude code. Accessed: 2025-09-29. find
  2. Giuseppe Attanasio, Beatrice Savoldi, Dennis Fucci, and Dirk Hovy. 2024. Twists, humps, and pebbles: Multilingual speech recognition models exhibit gender performance gaps. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21318–21340, Miami, Florida, U… find
  3. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. 2022. Xls-r: Self-supervised cross-lingual speech representation learning at scale. In Interspeech 2022…
  4. Catarina Belém, Preethi Seshadri, Yasaman Razeghi, and Sameer Singh. 2024. Are models biased on text without gender-related language? In International Conference on Representation Learning, volume 2024, pages 12876–12915. find
  5. Jad Bendarkawi, Ashley Ponce, Sean Chidozie Mata, Aminah Aliu, Yuhan Liu, Lei Zhang, Amna Liaqat, Varun Nagaraj Rao, and Andrés Monroy-Hernández. 2025. Conversar: Exploring embodied llm-powered group conversations in augmented reality for second language learners. In Proceedings of the Extended Abst… find
  6. May Pik Yu Chan, June Choe, Aini Li, Yiran Chen, Xin Gao, and Nicole Holliday. 2022. Training and typological bias in ASR performance for world Englishes. In Interspeech 2022, pages 1273–1277. find
  7. Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. 2024. Voicebench: Benchmarking llm-based voice assistants. Preprint, arXiv:2410.17196.
  8. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 other…
  9. Hend ElGhazaly, Bahman Mirheidari, Nafise Sadat Moosavi, and Heidi Christensen. 2025. Exploring gender disparities in automatic speech recognition technology. Preprint, arXiv:2502.18434. — [arXiv:2502.18434](https://arxiv.org/abs/2502.18434) find
  10. Rao Ma, Mengjie Qian, Siyuan Tang, Stefano Bannò, Kate M. Knill, and Mark J. F. Gales. 2025. Assessment of l2 oral proficiency using speech large language models. Preprint, arXiv:2505.21148. — [arXiv:2505.21148](https://arxiv.org/abs/2505.21148) find
  11. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. find
  12. Meta AI. 2025. The llama 4 herd: The beginning of a new era of natively multimodal intelligence. find
  13. Google Gemini Team. 2023. Gemini: A family of highly capable multimodal models. ArXiv, abs/2312.11805.
  14. Calbert Graham and Nathan Roll. 2024. Evaluating openai’s whisper asr: Performance analysis across diverse accents and speaker traits. JASA Express Letters, 4(2):025206. find
  15. Camille Harris, Chijioke Mgbahurike, Neha Kumar, and Diyi Yang. 2024. Modeling gender and dialect bias in automatic speech recognition. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15166–15184, Miami, Florida, USA. Association for Computational Linguistics.
  16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.
  17. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. 2024. Dialect prejudice predicts ai decisions about people’s character, employability, and criminality. Preprint, arXiv:2403.00742.
  18. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684–7689. find
  19. Ajinkya Kulkarni, Atharva Kulkarni, Miguel Couceiro, and Isabel Trancoso. 2024. Unveiling biases while embracing sustainability: Assessing the dual challenges of automatic speech recognition systems. In Interspeech 2024, interspeech 2024, pages 4628– 4632. ISCA. find
  20. CHENG LI, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang. 2024. Culturepark: Boosting cross-cultural understanding in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  21. Alexander H. Liu, Andy Ehrenberg, Andy Lo, Clément Denoix, Corentin Barreau, Guillaume Lample, Jean-Malo Delignon, Khyathi Raghavi Chandu, Patrick von Platen, Pavankumar Reddy Muddireddy, Sanchit Gandhi, Soham Ghosh, Srijan Mishra, Thomas Foubert, Abhinav Rastogi, Adam Yang, Albert Q. Jiang, Alexand… find
  22. Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, and 57 others. 2025. Phi-4-mini technical…
  23. Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. 2024. Spoken question answering and speech continuation using spectrogram-powered LLM. In The Twelfth International Conference on Le… find
  24. Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Associatio…
  25. OpenAI. 2022. Introducing ChatGPT. openai.com/blog/chatgpt. https:// find
  26. OpenAI. 2024. GPT-4o System Card. arXiv preprint. ArXiv:2410.21276.
  27. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. arXiv preprint.
  28. Resemble AI. 2025. Chatterbox-TTS. https://github.com/resemble-ai/chatterbox. GitHub repository. ↗
  29. Sujoy Roychowdhury, Ranjani H.G., Sumit Soman, Nishtha Paul, Subhadip Bandyopadhyay, and Siddhanth Iyengar. 2025. Intelligibility of Text-to-Speech Systems for Mathematical Expressions. In Interspeech 2025, pages 2280–2284. find
  30. Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield, James Qin, Danny Rozenberg, Tara Sainath, Johan Schalkwyk, Matt Sharifi, Michelle Tadmor Rama… — AudioPaLM: A Large Language Model That Can Speak and Listen
  31. C. E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423. find
  32. First ACL Workshop on Ethics in Natural Language Processing, pages 53–59, Valencia, Spain. Association for Computational Linguistics. Min-Han Shih, Ho-Lam Chung, Yu-Chi Pai, Ming-Hao Hsu, Guan-Ting Lin, Shang-Wen Li, and Hung yi Lee. 2024. GSQA: An End-to-End Model for Generative Spoken Question Ans… find
  33. Omnilingual ASR team, Gil Keren, Artyom Kozhevnikov, Yen Meng, Christophe Ropers, Matthew Setzler, Skyler Wang, Ife Adebara, Michael Auli, Can Balioglu, Kevin Chan, Chierh Cheng, Joe Chuang, Caley Droof, Mark Duppenthaler, Paul-Ambroise Duquenne, Alexander Erben, Cynthia Gao, Gabriel Mejia Gonzalez,… — [arXiv:2511.09690](https://arxiv.org/abs/2511.09690) find
  34. Hari Shrawgi, Prasanjit Rath, Tushar Singhal, and Sandipan Dandapat. 2024. Uncovering stereotypes in large language models: A task complexity-based approach. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages… find
  35. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Chosh… — Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
  36. Divya Tadimeti, Kallirroi Georgila, and David Traum. 2022. Evaluation of off-the-shelf speech recognizers on different accents in a dialogue domain. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6001–6008, Marseille, France. European Language Resources Associat… find
  37. Weiting Tan, Hirofumi Inaguma, Ning Dong, Paden D. Tomasello, and Xutai Ma. 2025. SSR: Alignment-aware modality connector for speech language models. In Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025), pages 56–75, Vienna, Austria (in-person and online).… find
  38. An Vo, Mohammad Reza Taesiri, Daeyoung Kim, and Anh Totti Nguyen. 2025. B-score: Detecting biases in large language models using response history. In Forty-second International Conference on Machine Learning. find
  39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
  40. Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2024. Unveiling selection biases: Exploring order and token sensitivity in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5598– 5621, Bangkok, Thailand. Association for Computation… find
  41. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15757–15773, Singapore. Associa…
  42. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. 2024. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations.
  43. Yixuan Tang and Anthony K.H: Tung. 2023. Squad-src: A dataset for multi-accent spoken reading comprehension. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 5206–5214. International Joint Conferences on Artificial Intelligence Organizati… find
  44. Rachael Tatman. 2017. Gender and dialect bias in YouTube’s automatic captions. In Proceedings of the

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wei-2026-bias-ear-listener-assessing-sensitivity/