Elamur Research⌕
☰
home / papers / 2026 / Closing the Modality Reasoning Gap for Speech Large Language Models

Closing the Modality Reasoning Gap for Speech Large Language Models

Paper · 2026 · Chaoren Wang, Heng Lu, Xueyao Zhang et al.

Summary

Speech LLMs, which pair a speech encoder and adapter with a text LLM, reason noticeably worse on spoken questions than on the same questions in text. The authors call this the modality reasoning gap. They attribute it to hidden-state drift across Transformer layers and to divergent behaviour during long reasoning chains. Earlier fixes either align only the input projector or use rigid, off-policy token-level distillation. The authors instead propose TARS (Trajectory Alignment for Reasoning in Speech), an on-policy reinforcement-learning method built on GRPO. For speech-conditioned outputs, TARS adds two dense rewards to the usual accuracy and format reward:

  • Representation alignment: layer-wise cosine similarity between speech and text hidden states.
  • Behavior alignment: semantic consistency between the output and the text-conditioned completion.

The text branch keeps training on the base reward and serves as a moving reference. The alignment rewards still give a useful signal when every sampled answer in a group is wrong. Tested on Phi-4-MM and Qwen2.5-Omni, TARS recovers roughly full text-level performance on speech. It reaches state-of-the-art results among 7B Speech LLMs on MMSU and OBQA without changing the architecture.

Key points

  • The gap is measured with the Modality Recovery Rate (MRR), defined as speech accuracy divided by the base model's text accuracy. The goal is MRR ≥ 100%. Reward weights are α = β = 1.0, and answers are extracted with xFinder.
  • Phi-4-MM:
    • TARS reaches 79.57% speech accuracy and an MRR of 100.28%, compared with 76.86% and 96.86% for text-only GRPO.
    • Text accuracy is essentially unchanged (83.56% vs. 83.76%).
  • Qwen2.5-Omni: the behavior reward alone works best (77.71% audio accuracy, MRR 100.01%). Adding the layer-wise reward over-regularizes this already heavily post-trained model.
  • Generalization: on real speech and paralinguistic tasks, TARS scores 57.40% on the paralinguistic MMSU benchmark, where CoT prompting drops to 51.50%. It scores 41.95% on the audio version of SD-QA.
  • Cost and availability: compute overhead is about 4.4% over standard GRPO. Code and data are released at github.com/AmphionTeam/TARS.
  • Limitations: the method is tested only at 7B scale and only for single-turn reasoning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓Chinese University of Hong Kong, ShenzhenShenzhen · China47 works · 2,596 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Chaoren Wang, Heng Lu, Xueyao Zhang, Shujie Liu, Yan Lu, Jinyu Li and Zhizheng Wu. 2026. Closing the Modality Reasoning Gap for Speech Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.857

@inproceedings{wang2026closing,
  title = {{Closing the Modality Reasoning Gap for Speech Large Language Models}},
  author = {Chaoren Wang, Heng Lu, Xueyao Zhang, Shujie Liu, Yan Lu, Jinyu Li and Zhizheng Wu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.857},
  url = {https://doi.org/10.18653/v1/2026.acl-long.857}
}

References

46 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaoka… — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  2. Umberto Cappellazzo, Enrico Fini, Muqiao Yang, Daniele Falavigna, Alessio Brutti, and Bhiksha Raj. 2024. Continual contrastive spoken language understanding. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3727– 3741, Bangkok, Thailand. Association for Computational Lin… find
  3. Wei Chen, Yunzhong Zheng, Tu Zheng, Deng Cai, Wenxiao Wang, and Jieping Ye. 2026. Alignchat: Endowing LLMs with end-to-end speech-to-text chat capability through token-level representation alignment. find
  4. Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T Tan, and Haizhou Li. 2024. Voicebench: Benchmarking llm-based voice assistants. arXiv preprint arXiv:2410.17196.
  5. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919.
  6. Santiago Cuervo, Skyler Seto, Maureen de Seyssel, Richard He Bai, Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly, and Zakaria Aldeneh. 2025. Closing the gap between text and speech understanding in llms. arXiv preprint arXiv:2510.13632. — [arXiv:2510.13632](https://arxiv.org/abs/2510.13632) find
  7. Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Steven Y Guo, and Irwin King. 2025. Recent advances in speech language models: A survey. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13943–139…
  8. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… — SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Models
  9. Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. 2024. Cosyvoice 2: Scalable streaming speech synthesis with large langua… find
  10. Fahim Faisal, Sharlina Keshava, Md Mahfuz Ibn Alam, and Antonios Anastasopoulos. 2021. SD-QA: Spoken dialectal question answering for the real world. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3296–3315, Punta Cana, Dominican Republic. Association for Computation… find
  11. Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. STEMM: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7050–7062, Dublin, Ireland. Associ…
  12. Yuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky, and James Glass. 2023. Listen, think, and understand. arXiv preprint arXiv:2305.10790. — [arXiv:2305.10790](https://arxiv.org/abs/2305.10790) find
  13. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations. find
  14. Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. 2025. Emilia: A large-scale, extensive, multilingual, and diverse dataset for speech generation. IEEE Transactions on Audi… find
  15. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700.
  16. KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, Jun Chen, Yanru Chen, Yulun Du, Weiran He, Zhenxing Hu, Guokun Lai, Qingcheng Li, Y… — Kimi-Audio Technical Report
  17. Pengcheng Li, Botao Zhao, Zuheng Kang, Junqing Peng, Xiaoyang Qu, Yayun He, and Jianzong Wang. 2025. Emo-rl: Emotion-rule-based reinforcement learning enhanced audio-language model for generalized speech emotion recognition. In Findings of the Association for Computational Linguistics: EMNLP 2025, p… find
  18. Keliang Liu, Dingkang Yang, Ziyun Qian, Weijie Yin, Yuchi Wang, Hongsheng Li, Jun Liu, Peng Zhai, Yang Liu, and Lihua Zhang. 2025a. Reinforcement learning meets large language models: A survey of advancements and applications across the llm lifecycle. arXiv preprint arXiv:2509.16679. — [arXiv:2509.16679](https://arxiv.org/abs/2509.16679) find
  19. Yansong Liu, Jiateng Li, and Yuan Liu. 2025b. Enhancing speech large language models through reinforced behavior alignment. arXiv preprint arXiv:2509.03526. — [arXiv:2509.03526](https://arxiv.org/abs/2509.03526) find
  20. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025c. Understanding r1-zero-like training: A critical perspective. In Second Conference on Language Modeling.
  21. Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Jagadeesh Balam, Boris Ginsburg, Yu-Chiang Frank Wang, and Hung-yi Lee. 2025a. Developing instruction-following speech language model without speech instruction-tuning data. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speec… find
  22. Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Chih-Kai Yang, Chee-En Yu, Chun-Wei Chen, Wei-Chih Chen, Chien-yu Huang, Yi-Cheng Lin, Yu-Xiang Lin, Chi-An Fu, Chun-Yi Kuan, Wenze Ren, Xuanjun Chen, Wei-Ping Huang, En-Pei Hu, Tzu-Quan Lin, Yuan-Kuei Wu, Kuan-Po Huang, Hsiao… — [arXiv:2507.02768](https://arxiv.org/abs/2507.02768) find
  23. Pooneh Mousavi, Yingzhi Wang, Mirco Ravanelli, and Cem Subakan. 2025. Alas: Measuring latent speech-text alignment for spoken language understanding in multimodal llms. arXiv preprint arXiv:2505.19937. — [arXiv:2505.19937](https://arxiv.org/abs/2505.19937) find
  24. Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, and Emmanuel Dupoux. 2025. SpiRit-LM: Interleaved spoken and…
  25. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  26. Jing Peng, Yucheng Wang, Bohan Li, Yiwei Guo, Hankun Wang, YanGui Fang, Yu Xi, Haoyu Li, Xu Li, Ke Zhang, Shuai Wang, and Kai Yu. 2025a. A survey on speech large language models for understanding. IEEE Journal of Selected Topics in Signal Processing, pages 1–32. find
  27. Yifan Peng, Krishna C Puvvada, Zhehuai Chen, Piotr Zelasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, and Boris Ginsburg. 2025b. VoiceTextBlender: Augmenting large language models with speech capabilities via single-stage joint speech-text supervised fine-tuning. In Proceeding… find
  28. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. find
  29. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  30. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  31. Weiting Tan, Hirofumi Inaguma, Ning Dong, Paden D. Tomasello, and Xutai Ma. 2025. SSR: Alignment-aware modality connector for speech language models. In Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025), pages 56–75, Vienna, Austria (in-person and online).… find
  32. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289.
  33. Dingdong WANG, Junan Li, Jincenzi Wu, Dongchao Yang, Xueyuan Chen, Tianhua Zhang, and Helen M. Meng. 2026. MMSU: A massive multi-task spoken language understanding and reasoning benchmark. In The Fourteenth International Conference on Learning Representations. find
  34. Enzhi Wang, Qicheng Li, Zhiyuan Tang, and Yuhang Jia. 2025. Cross-modal knowledge distillation for speech large language models. arXiv preprint arXiv:2509.14930. — [arXiv:2509.14930](https://arxiv.org/abs/2509.14930) find
  35. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark…
  36. Bajian Xiang, Shuaijiang Zhao, Tingwei Guo, and Wei Zou. 2025. Understanding the modality gap: An empirical study on the speech-text alignment mechanism of large speech language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5187–5202, Suzhou… find
  37. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025a. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. — Qwen2.5-Omni Technical Report
  38. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu,… — Qwen3-Omni Technical Report
  39. Wenze Xu, Chun Wang, Jiazhen Yu, Sheng Chen, Liang Gao, and Weihong Deng. 2025c. Optimal transport regularization for speech text alignment in spoken language models. In Asian Conference on Pattern Recognition, pages 280–294. Springer. find
  40. Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. Minicpm-v: A gpt-4…
  41. Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu li, Feiyu Xiong, Bo Tang, and Ding Chen. 2025a. xfinder: Large language models as automated evaluators for reliable evaluation. In International Conference on Representation Learning, volume 2025, pages 59850–59892. find
  42. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli… — DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  43. LLM-Core-Team Xiaomi. 2025. Mimo-audio: Audio language models are few-shot learners.
  44. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176.
  45. Jingran Xie, Xiang Li, Hui Wang, Yue Yu, Yang Xiang, Xixin Wu, and Zhiyong Wu. 2025. Enhancing generalization of speech large language models with multi-task behavior imitation and speech-text interleaving. arXiv preprint arXiv:2505.18644. — [arXiv:2505.18644](https://arxiv.org/abs/2505.18644) find
  46. Maike Züfle and Jan Niehues. 2025. Contrastive learning for task-independent SpeechLLM-pretraining. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8469–8490, Vienna, Austria. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-closing-modality-reasoning-gap-speech/