Elamur Research⌕
☰
home / papers / 2026 / Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps

Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps

Paper · 2026 · Jonas Waldendorf, Bashar Awwad Shiekh Hasan and Evgenii Tsymbalov

Summary

SpeechLLMs sometimes produce fluent output that is not supported by the audio. Most existing ways to catch these hallucinations need reference transcripts, which are not available during deployment. The authors propose a reference-free detector that runs at inference time and reads the model's own attention maps. They define four attention features: AUDIO RATIO, AUDIO CONSISTENCY, AUDIO ENTROPY and TEXT ENTROPY. Each is computed per head and averaged over decoding steps, and the results feed simple logistic regression classifiers. The features target two failure signs: the diagonal audio–text alignment breaks down, and attention slides back to the first audio frames.

The method was tested on Qwen-2-Audio and Voxtral-3B for ASR (VoxPopuli, CALLHOME) and speech-to-text translation (FLEURS). It beats uncertainty-based and earlier attention-based baselines in-domain, and it transfers to out-of-domain ASR. About 100 selected heads perform as well as the full set and generalise better. However, detectors trained on one task do not transfer to the other, so each task needs its own training data. The work shows that internal attention offers a cheap, practical signal for filtering unreliable speech output.

Key points

  • Largest gain: up to +0.23 PR-AUC over baselines, reached on in-domain VoxPopuli ASR with Voxtral-3B.
  • Evaluation data:
    • VoxPopuli: English, German, French and Spanish combined, 7,080 sentences.
    • CALLHOME: 3,916 examples after filtering.
    • FLEURS S2TT: 4,613 examples.
  • Labelling: 1,950 human-annotated examples were used to calibrate thresholds for automatic labelling. The resulting labels have precision 0.979 but recall of only 0.443.
  • Translation results: baselines are weak on S2TT, with a best PR-AUC of 0.25 for Qwen-2-Audio and 0.17 for Voxtral-3B. Training on S2TT data lifts Qwen-2-Audio to 0.44 PR-AUC (Top 150 heads), and combining all four features matters more for translation than for ASR.
  • Head selection: the top-50 heads chosen for ASR and S2TT overlap little, from 8% to 32% depending on the metric, with AUDIO CONSISTENCY the most stable. This points to task-specific hallucination signals. Voxtral-3B also generalises across domains better than Qwen-2-Audio.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Jonas Waldendorf, Bashar Awwad Shiekh Hasan and Evgenii Tsymbalov. 2026. Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2147

@inproceedings{waldendorf2026detecting,
  title = {{Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps}},
  author = {Jonas Waldendorf, Bashar Awwad Shiekh Hasan and Evgenii Tsymbalov},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2147},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2147}
}

References

31 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hanin Atwany, Abdul Waheed, Rita Singh, Monojit Choudhury, and Bhiksha Raj. 2025. Lost in transcription, found in distribution shift: Demystifying hallucination in speech foundation models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 23181–23203, Vienna, Austria. As… find
  2. Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, and 1 others. 2023. Seamlessm4t: massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596.
  3. Alexandra Canavan, David Graff, and George Zipperlen. 1997. CALLHOME American English Speech. LDC97S42, Web Download. find
  4. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. Qwen2-audio technical report. Preprint, arXiv:2407.10759.
  5. Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, and James R. Glass. 2024. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proces…
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  7. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. Fleurs: Few-shot learning evaluation of universal representations of speech. arXiv preprint arXiv:2205.12446.
  8. Rita Frieske and Bertram E. Shi. 2024. Hallucinations in neural automatic speech recognition: Identifying errors and hallucinatory models. Preprint, arXiv:2401.01572. — [arXiv:2401.01572](https://arxiv.org/abs/2401.01572) find
  9. Nuno M. Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023a. Hallucinations in large multilingual translation models. Transactions of the Association for Computational Linguistics, 11:1500–1517.
  10. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xCOMET: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995.
  11. Nuno M. Guerreiro, Elena Voita, and André Martins. 2023b. Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1059–1075, Dubrov…
  12. Matthew B Hoy. 2018. Alexa, siri, cortana, and more: an introduction to voice assistants. Medical reference services quarterly, 37(1):81–88. find
  13. Hsiu-Yuan Huang, Yutong Yang, Zhaoxi Zhang, Sanwoo Lee, and Yunfang Wu. 2024. A survey of uncertainty estimation in llms: Theory meets practice. Preprint, arXiv:2410.15326. — [arXiv:2410.15326](https://arxiv.org/abs/2410.15326) find
  14. Alkis Koudounas, Moreno La Quatra, Manuel Giollo, Sabato Marco Siniscalchi, and Elena Baralis. 2025. Hallucination benchmark for speech foundation models. Preprint, arXiv:2510.16567. — [arXiv:2510.16567](https://arxiv.org/abs/2510.16567) find
  15. Moritz Laurer, Wouter van Atteveldt, Andreu Salleras Casas, and Kasper Welbers. 2022. Less Annotating, More Classifying – Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT - NLI. Preprint. Publisher: Open Science Framework. find
  16. Alexander H. Liu, Andy Ehrenberg, Andy Lo, Clément Denoix, Corentin Barreau, Guillaume Lample, Jean-Malo Delignon, Khyathi Raghavi Chandu, Patrick von Platen, Pavankumar Reddy Muddireddy, Sanchit Gandhi, Soham Ghosh, Srijan Mishra, Thomas Foubert, Abhinav Rastogi, Adam Yang, Albert Q. Jiang, Alexand… find
  17. Andrey Malinin and Mark John Francis Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations. find
  18. Andrey Malinin, Anton Ragni, Kate Knill, and Mark Gales. 2017. Incorporating uncertainty into deep learning for spoken language assessment. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 45–50, Vancouver, Canada. Association… find
  19. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  20. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  21. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  22. Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1172–1183, Online.…
  23. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  24. Elena Voita, Rico Sennrich, and Ivan Titov. 2021. Analyzing the source and target contributions to predictions in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro… find
  25. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Annual…
  26. Yingzhi Wang, Anas Alhmoud, Saad Alsahly, Muhammad Alqurishi, and Mirco Ravanelli. 2025. Calm-whisper: Reduce whisper hallucination on non-speech by calming crazy heads down. ArXiv, abs/2505.12969. — [arXiv:2505.12969](https://arxiv.org/abs/2505.12969) find
  27. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567.
  28. Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. 2024a. Llm-check: Investigating detection of hallucinations in large language models. In Advances in Neural Information Processing Systems, volume 37, pages 34188–34216. Curran Associate… find
  29. Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. 2024b. LLM-check: Investigating detection of hallucinations in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  30. Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2025. Benchmarking uncertainty quantificat…
  31. Artem Vazhentsev, Lyudmila Rvanova, Gleb Kuzmin, Ekaterina Fadeeva, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Mrinmaya Sachan, Preslav Nakov, and Artem Shelmanov. 2025. Uncertainty-aware attention heads: Efficient unsupervised uncertainty quantification for llms. Preprint, ar… — [arXiv:2505.20045](https://arxiv.org/abs/2505.20045) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/waldendorf-2026-detecting-hallucinations-speechllms-inference-time/