Elamur Research⌕
☰
home / papers / 2026 / Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception

Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception

Paper · 2026 · Zhen Wan, Chao-Han Huck Yang, Jinchuan Tian et al.

Summary

Omni-modal models that process audio and text should, in principle, improve speech recognition when also given transcripts from an external ASR system. The authors show the opposite. When Qwen2.5-Omni is fine-tuned to correct 5-best hypotheses from Whisper-v2-large, WER gets worse than either system alone, and changing the prompt does not fix it. The cause is that the model has no way to decide which source to believe, so flawed external candidates mislead it.

Speech-Hands addresses this by making the decision explicit. Before producing its answer, the model emits an action token:

  • trust its own first-pass output,
  • adopt the external system's output, or
  • rewrite a new answer from all available evidence.

Training labels for these tokens come from checking which option actually scores best on each example. For ASR, that comparison uses WER.

The same mechanism transfers from transcription to multiple-choice audio question answering. It gives consistent gains on OpenASR benchmarks and strong audio QA accuracy. The token choices also make the model's decisions interpretable and more robust.

Key points

  • Preliminary negative result: naive fine-tuning for generative error correction (GER) on OpenASR data raised average WER.
    • The baselines were 7.33 for Whisper and 7.17 for Qwen-omni.
    • GER with text only reached about 8.44, and GER with text plus audio about 8.07.
    • An oracle that always picks the better output would reach 5.00.
    • Prompt variants still gave 8.52–9.05.
  • Action tokens: the model chooses among <internal>, <external> and <rewrite>. A fourth action, tool calling, is proposed but left for future work.
  • Reported results:
    • a 12.1% WER improvement over strong baselines across seven OpenASR benchmarks;
    • 77.37% accuracy on audio QA, with high F1 for the action decisions.
  • Rewrite behaviour: <rewrite> labels are rare in training, but when the model triggers it, precision is high and exceeds recall.
  • Limitations:
    • ASR training used only a restricted data subset.
    • The system accepts only one external model.
    • Transfer to a different external ASR system at test time is not studied.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓Kyoto UniversityKyoto · Japan40 works · 8,235 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksSpeech & spoken language · 480 works

Cite

Cite as Zhen Wan, Chao-Han Huck Yang, Jinchuan Tian, Hanrong Ye, Ankita Pasad, Szu-Wei Fu, Arushi Goel, Ryo Hachiuma, Shizhe Diao, Kunal Dhawan, Sreyan Ghosh, Yusuke Hirota, Zhehuai Chen, Rafael Valle, Chenhui Chu, Shinji Watanabe, Boris Ginsburg and Yu-Chiang Frank Wang. 2026. Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1997

@inproceedings{wan2026speech,
  title = {{Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception}},
  author = {Zhen Wan, Chao-Han Huck Yang, Jinchuan Tian, Hanrong Ye, Ankita Pasad, Szu-Wei Fu, Arushi Goel, Ryo Hachiuma, Shizhe Diao, Kunal Dhawan, Sreyan Ghosh, Yusuke Hirota, Zhehuai Chen, Rafael Valle, Chenhui Chu, Shinji Watanabe, Boris Ginsburg and Yu-Chiang Frank Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1997},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1997}
}

References

51 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. 2025. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743.
  2. Sharath Adavanne, Archontis Politis, and Tuomas Virtanen. 2019. A multi-room reverberant dataset for sound event localization and detection. Preprint, arXiv:1905.08546. — [arXiv:1905.08546](https://arxiv.org/abs/1905.08546) find
  3. A. Calcus. 2024. Development of auditory scene analysis: a mini-review. Frontiers in Human Neuroscience, 18:1352247. find
  4. Jean Carletta. 2007. Unleashing the killer corpus: experiences in creating the multi-everything ami meeting corpus. Language Resources and Evaluation, 41(2):181–190. find
  5. Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan. 2021. Gigaspeech: An evolving, mu… find
  6. Yifu Chen, Shengpeng Ji, Haoxiao Wang, Ziqing Wang, Siyu Chen, Jinzheng He, Jin Xu, and Zhou Zhao. 2025. Wavrag: Audio-integrated retrieval augmented generation for spoken dialogue models. Preprint, arXiv:2502.14727.
  7. Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou, and Yang Liu. 2024. Vision-language models can self-improve reasoning via reflection. arXiv preprint arXiv:2411.00855.
  8. Yangui Fang, Baixu Cheng, Jing Peng, Xu Li, Yu Xi, Chengwei Zhang, and Guohui Zhong. 2025. Fewer hallucinations, more verification: A three-stage llm-based framework for asr error correction. arXiv preprint arXiv:2505.24347. — [arXiv:2505.24347](https://arxiv.org/abs/2505.24347) find
  9. B. Galantucci, C. A. Fowler, and M. T. Turvey. 2006. The motor theory of speech perception reviewed. Psychonomic Bulletin & Review, 13(3):361–377. Erratum in: Psychon Bull Rev. 2006 Aug;13(4):742. find
  10. Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. 2025. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. Preprint, arXiv:2507.08128. — [arXiv:2507.08128](https://arxiv.org/abs/2507.08128) find
  11. Arushi Goel, Karan Sapra, Matthieu Le, Rafael Valle, Andrew Tao, and Bryan Catanzaro. 2024. Omcat: Omni context aware transformer. arXiv preprint arXiv:2410.12109. — [arXiv:2410.12109](https://arxiv.org/abs/2410.12109) find
  12. Robie Gonzales and Frank Rudzicz. 2024. A retrieval augmented approach for text-to-music generation. In Proceedings of the 3rd Workshop on NLP for Music and Audio (NLP4MusA), pages 31–36, Oakland, USA. Association for Computational Lingustics. find
  13. François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Estève. 2018. TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation, page 198–208. Springer International Publishing. find
  14. Rui Hu, Delai Qiu, Shuyu Wei, Jiaming Zhang, Yining Wang, Shengping Liu, and Jitao Sang. 2025. Investigating and enhancing vision-audio capability in omnimodal large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 7452–7463, Vienna, Austria. Association… find
  15. Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. 2023. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. Preprint, arXiv:2301.12661. — [arXiv:2301.12661](https://arxiv.org/abs/2301.12661) find
  16. M. Kaiser, D. Senkowski, and J. Keil. 2021. Mediofrontal theta-band oscillations reflect top-down influence in the ventriloquist illusion. Human Brain Mapping, 42(2):452–466. find
  17. Yuma Koizumi, Yasunori Ohishi, Daisuke Niizumi, Daiki Takeuchi, and Masahiro Yasuda. 2020. Audio captioning using pre-trained large-scale language model guided by audio-based similar caption retrieval. Preprint, arXiv:2012.07331. — [arXiv:2012.07331](https://arxiv.org/abs/2012.07331) find
  18. C. Lebiere and J. R. Anderson. 2011. Cognitive Constraints on Decision Making under Uncertainty. Frontiers in Psychology, 2:305. find
  19. Gang Li, Jizhong Liu, Heinrich Dinkel, Yadong Niu, Xingwei Sun, Tianzi Wang, Junbo Zhang, and Jian Luan. 2025a. Miaqa submission for dcase 2025 challenge task 5: A reinforcement learning driven audio question answering method. Technical report, DCASE2025 Challenge. find
  20. Yadong Li, Jun Liu, Tao Zhang, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, Chong Li, et al. 2025b. Baichuan-omni-1.5 technical report. Preprint, arXiv:2501.15368.
  21. Yen-Ting Lin, Zhehuai Chen, Piotr Żelasko, Zhen Wan, Xuesong Yang, Zih-Ching Chen, Krishna C Puvvada, Ke Hu, Szu-Wei Fu, Jun Wei Chiu, et al. 2025. Neko: Cross-modality post-recognition error correction with tasks-guided mixture-of-experts language model. In Proceedings of the 63rd Annual Meeting o… find
  22. Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, et al. 2025. Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix. arXiv preprint arXiv:2505.13032. — [arXiv:2505.13032](https://arxiv.org/abs/2505.13032) find
  23. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. find
  24. Christian Marinoni, Riccardo Fosco Gramaccioni, Changan Chen, Aurelio Uncini, and Danilo Com-miniello. 2024. Overview of the l3das23 challenge on audio-visual extended reality. Preprint, arXiv:2402.09245. — [arXiv:2402.09245](https://arxiv.org/abs/2402.09245) find
  25. Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao,… — Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
  26. Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Mal… — GPT-4 Technical Report
  27. Thomas O. Nelson. 1990. Metamemory: A Theoretical Framework and New Findings. In Gordon H. Bower, editor, Psychology of Learning and Motivation, volume 26, pages 125–173. Academic Press. find
  28. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… find
  29. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. find
  30. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. Preprint, arXiv:2212.04356.
  31. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  32. Matthew Renze and Erhan Guven. 2024. Self-reflection in llm agents: Effects on problem-solving performance. arXiv preprint arXiv:2405.06682.
  33. S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2024a. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410.19168. find
  34. S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2024b. Mmau: A massive multi-task audio understanding and reasoning benchmark. Preprint, arXiv:2410.19168. find
  35. Laela Sayigh, Mary Ann Daher, Julie Allen, Helen Gordon, Katherine Joyce, Claire Stuhlmann, and Peter Tyack. 2016. The watkins marine mammal sound database: an online, freely accessible resource. In Proceedings of Meetings on Acoustics, volume 27, page 040013. Acoustical Society of America. find
  36. Monica Sekoyan, Nithin Rao Koluguri, Nune Tade-vosyan, Piotr Zelasko, Travis Bartley, Nikolay Karpov, Jagadeesh Balam, and Boris Ginsburg. 2025. Canary-1b-v2 & parakeet-tdt-0.6b-v3: Efficient and high-performance models for multilingual asr and ast. Preprint, arXiv:2509.14128. — [arXiv:2509.14128](https://arxiv.org/abs/2509.14128) find
  37. Robert L Selman and Diane F Byrne. 1974. A structural-developmental analysis of levels of role taking in middle childhood. Child development, pages 803– 806. find
  38. Zihan Song, Xin Wang, Zi Qian, Hong Chen, Longtao Huang, Hui Xue, and Wenwu Zhu. Modularized self-reflected video reasoner for multimodal llm with application to video question answering. In Forty-second International Conference on Machine Learning. find
  39. Zhen Wan, Chao-Han Huck Yang, Yahan Yu, Jinchuan Tian, Sheng Li, Ke Hu, Zhehuai Chen, Shinji Watanabe, Fei Cheng, Chenhui Chu, et al. 2025. Speechiq: Speech-agentic intelligence quotient across cognitive levels in voice understanding by large language models. In Proceedings of the 63rd Annual Meetin… find
  40. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Annual…
  41. Zixuan Wang, Chi-Keung Tang, and Yu-Wing Tai. 2025. Audio-agent: Leveraging llms for audio generation, editing and composition. Preprint, arXiv:2410.03335. — [arXiv:2410.03335](https://arxiv.org/abs/2410.03335) find
  42. Zhifei Xie and Changqiao Wu. 2024. Mini-omni: Language models can hear, talk while thinking in streaming. Preprint, arXiv:2408.16725. find
  43. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025a. Qwen2.5-omni technical report. Preprint, arXiv:2503.20215.
  44. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. 2025b. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. — Qwen2.5-Omni Technical Report
  45. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. 2025c. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765.
  46. Chao-Han Huck Yang, Sreyan Ghosh, Qing Wang, Jaeyeon Kim, Hengyi Hong, Sonal Kumar, Guirui Zhong, Zhifeng Kong, S Sakshi, Vaibhavi Lokegaonkar, Oriol Nieto, Ramani Duraiswami, Dinesh Manocha, Gunhee Kim, Jun Du, Rafael Valle, and Bryan Catanzaro. 2025a. Multi-domain audio question answering toward a… — [arXiv:2505.07365](https://arxiv.org/abs/2505.07365) find
  47. Chao-Han Huck Yang, Yile Gu, Yi-Chieh Liu, Shalini Ghosh, Ivan Bulyko, and Andreas Stolcke. 2023a. Generative speech recognition error correction with large language models and task-activating prompting. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. IEEE. find
  48. Qize Yang, Shimin Yao, Weixuan Chen, Shenghao Fu, Detao Bai, Jiaxing Zhao, Boyuan Sun, Bowen Yin, Xihan Wei, and Jingren Zhou. 2025b. Humanomniv2: From understanding to omni-modal reasoning with context. arXiv preprint arXiv:2506.21277. — [arXiv:2506.21277](https://arxiv.org/abs/2506.21277) find
  49. Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023b. Mm-react: Prompting chatgpt for multimodal reasoning and action. Preprint, arXiv:2303.11381. find
  50. Lu Zhang, Tiancheng Zhao, Heting Ying, Yibo Ma, and Kyusong Lee. 2024. Omagent: A multimodal agent framework for complex video understanding with task divide-and-conquer. Preprint, arXiv:2406.16620. — [arXiv:2406.16620](https://arxiv.org/abs/2406.16620) find
  51. Jinming Zhao, Gholamreza Haffari, and Ehsan Shareghi. 2023. Generating synthetic speech from SpokenVo-cab for speech translation. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1975–1981, Dubrovnik, Croatia. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wan-2026-speech-hands-self-reflection-voice/