Elamur Research⌕
☰
home / papers / 2026 / Afrispeech Semantics: Evaluating Audio–Semantic Reasoning in Spoken Language Models Across Domains and Accents

Afrispeech Semantics: Evaluating Audio–Semantic Reasoning in Spoken Language Models Across Domains and Accents

Paper · 2026 · Chibuzor Okocha and Christan Grant

Summary

Audio language models (ALMs) are usually benchmarked on transcription, captioning, retrieval or QA. These tests reveal little about whether a model draws conclusions strictly from what it hears. The authors build a multi-task benchmark on African-accented English speech. It probes five audio-grounded reasoning abilities: entailment and consistency, judging plausibility, and two accent-focused checks known as drift and restraint.

Each item pairs an audio premise with a textual hypothesis. The hypotheses were drafted with a LLaMA model and then checked and corrected by human annotators. Seven next-token-prediction ALMs and three contrastive CLAP-style models were evaluated zero-shot under a single shared protocol.

Generative models clearly outperform contrastive ones, which sit near chance. Even the generative models over-entail, accept hypotheses that are merely plausible, and give different judgments depending on accent and domain. These errors appear even when transcription is good.

The work matters because standard benchmarks likely understate reasoning failures in ALMs. It also offers a route toward fairer evaluation across accents and domains.

Key points

  • Data: four AfriSpeech-derived corpora from intronhealth on Hugging Face.
    • AfriSpeech-200: ~200 h
    • AfriSpeech-General (afrispeech-dialog): ~2.07 h
    • Afri-Names: ~8.92 h
    • Afrispeech-Medical (med-convo-nig): ~4.20 h
    • Accent counts in the overview table are 13, 8, 12 and 6.
  • Tasks:
    • Entailment uses three labels (E/N/C).
    • Consistency is binary, with no neutral option.
    • Plausibility tests whether models accept claims that fit commonsense but are not stated in the audio.
    • Accent drift and accent restraint check that judgments stay the same when only the accent changes.
  • Models evaluated:
    • Generative: AudioFlamingo2, AudioFlamingo3, GAMA, Kimi, Qwen2.5 Omni, Qwen2 Audio 7B, SALMONN.
    • Contrastive: LAION-CLAP, MSCLAP_22, MSCLAP_23.
  • Entailment results (as parsed from the extracted tables):
    • On Afri-200, Qwen2 Audio 7B appears to score highest (~0.71 accuracy).
    • On Medical, Kimi appears to lead (~0.65).
    • Contrastive models stay around 0.30–0.38 accuracy.
  • Stated limitations:
    • Coverage of accents and domains is finite.
    • Annotator judgments are subjective.
    • LLM-generated hypotheses may carry biases.
    • Only zero-shot settings were tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of FloridaGainesville · United States33 works · 364 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksSpeech & spoken language · 480 works

Cite

Cite as Chibuzor Okocha and Christan Grant. 2026. Afrispeech Semantics: Evaluating Audio–Semantic Reasoning in Spoken Language Models Across Domains and Accents. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.343

@inproceedings{okocha2026afrispeech,
  title = {{Afrispeech Semantics: Evaluating Audio–Semantic Reasoning in Spoken Language Models Across Domains and Accents}},
  author = {Chibuzor Okocha and Christan Grant},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.343},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.343}
}

References

27 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin, Linjie Li, Radu Kopetz, Yao Qian, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, and Lijuan Wang. 2025. Audio-Aware Large Language Models as Judges for Speaking Styles. arXiv preprint. ArXiv:2506.05984 [eess]. find
  2. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. Qwen2-audio technical report. Preprint, arXiv:2407.10759.
  3. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models. arXiv preprint. ArXiv:2311.07919 [eess].
  4. Soham Deshmukh, Dareen Alharthi, Benjamin Elizalde, Hannes Gamper, Mahmoud Al Ismail, Rita Singh, Bhiksha Raj, and Huaming Wang. 2024. PAM: Prompting Audio-Language Models for Audio Quality Assessment. In Interspeech 2024, pages 3320– 3324. ISCA. find
  5. Soham Deshmukh, Benjamin Elizalde, Dimitra Emmanouilidou, Bhiksha Raj, Rita Singh, and Huaming Wang. 2023a. Training Audio Captioning Models without Audio. arXiv preprint. ArXiv:2309.07372 [eess]. — [arXiv:2309.07372](https://arxiv.org/abs/2309.07372) find
  6. Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. 2023b. Pengi: An Audio Language Model for Audio Tasks. Advances in Neural Information Processing Systems, 36:18090–18108. find
  7. Soham Deshmukh, Shuo Han, Hazim Bukhari, Benjamin Elizalde, Hannes Gamper, Rita Singh, and Bhiksha Raj. Audio Entailment: Assessing Deductive Reasoning for Audio Understanding. find
  8. Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023. CLAP Learning Audio Concepts from Natural Language Supervision. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. ISSN: 2379-190X. find
  9. Kuofeng Gao, Shu-Tao Xia, Ke Xu, Philip Torr, and Jindong Gu. 2025. Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4763–4784, Vienna, Austr…
  10. Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S. Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024. GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities. arXiv preprint. ArXiv:2406.11768 [cs].
  11. Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. 2025. Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models. arXiv preprint. ArXiv:2507.081… — [arXiv:2507.08128](https://arxiv.org/abs/2507.08128) find
  12. KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, and 21 others. 2025. Kimi-Audio Technical Report. arXiv preprint. ArXiv:2504.18425…
  13. Marek Kubis, Paweł Skórzewski, Iwona Christop, Mateusz Czyżnikiewicz, Jakub Kubiak, Łukasz Bondaruk, and Marcin Lewandowski. 2025. Preservation of Language Understanding Capabilities in Speech-aware Large Language Models. arXiv preprint. ArXiv:2509.12171 [cs]. — [arXiv:2509.12171](https://arxiv.org/abs/2509.12171) find
  14. Tobi Olatunji, Tejumade Afonja, Aditya Yadavalli, Chris Chinenye Emezue, Sahib Singh, Bonaventure F. P. Dossou, Joanne Osuchukwu, Salomey Osei, Atnafu Lambebo Tonja, Naome Etori, and Clinton Mbataku. 2023. AfriSpeech-200: Pan-African accented speech dataset for clinical and general domain ASR. Trans…
  15. OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, A. J. Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, and 400 others. 2024. GPT-4o…
  16. Jing Peng, Yucheng Wang, Yu Xi, Xu Li, Xizhuo Zhang, and Kai Yu. 2025. A Survey on Speech Large Language Models. arXiv preprint. ArXiv:2410.18908 [eess] version: 3. — [arXiv:2410.18908](https://arxiv.org/abs/2410.18908) find
  17. Vinu Sankar Sadasivan, Soheil Feizi, Rajiv Mathews, and Lun Wang. 2025. Attacker’s Noise Can Manipulate Your Audio-based LLM in the Real World. arXiv preprint. ArXiv:2507.06256 [cs].
  18. S. Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2024. MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark. arXiv preprint. ArXiv:2410.19168 [eess]. find
  19. Mardhiyah Sanni, Tassallah Abdullahi, Devendra D. Kayande, Emmanuel Ayodele, Naome A. Etori, Michael S. Mollel, Moshood Yekini, Chibuzor Okocha, Lukman E. Ismaila, Folafunmi Omofoye, Boluwatife A. Adewale, and Tobi Olatunji. 2025. Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conver…
  20. Jiatong Shi, Shih-Heng Wang, William Chen, Martijn Bartelds, Vanya Bannihatti Kumar, Jinchuan Tian, Xuankai Chang, Dan Jurafsky, Karen Livescu, Hung-yi Lee, and Shinji Watanabe. 2024. ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets. arXiv p… find
  21. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289.
  22. Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F Chen. 2024. Audiobench: A universal benchmark for audio large language models. arXiv preprint arXiv:2406.16020.
  23. Qing Wang, Zehan Li, Hang Lv, Hongjie Chen, Yaodong Song, Jian Kang, Jie Lian, Jie Li, Yongxiang Li, Zhongjiang He, and Xuelong Li. 2025. BoSS: Beyond-Semantic Speech. arXiv preprint. ArXiv:2507.17563 [cs]. — [arXiv:2507.17563](https://arxiv.org/abs/2507.17563) find
  24. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. 2024. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension. In Proceedings of the 62nd Annual Meeting of the Association for Computati…
  25. Wanqi Yang, Yanda Li, Yunchao Wei, Meng Fang, and Ling Chen. 2025a. SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models. arXiv preprint. ArXiv:2508.02018 [cs] version: 1. — [arXiv:2508.02018](https://arxiv.org/abs/2508.02018) find
  26. Wanqi Yang, Yanda Li, Yunchao Wei, Meng Fang, and Ling Chen. 2025b. SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models. arXiv preprint. ArXiv:2508.02018 [cs] version: 1. — [arXiv:2508.02018](https://arxiv.org/abs/2508.02018) find
  27. Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. Salmonn-omni: A speech understanding and generation llm in a codec-free full-duplex framework. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/okocha-2026-afrispeech-semantics-evaluating-audio-semantic/