Elamur Research⌕
☰
home / papers / 2026 / Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs

Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs

Paper · 2026 · Linhao Zhang, Yuhan Song, Aiwei Liu et al.

Summary

Audio LLMs often handle reasoning benchmarks well but struggle with basic acoustic perception, such as identifying speaker traits, emotion, prosody or background sounds. The authors trace this to ASR-centric training. Transcripts reward recovering the words and implicitly teach models to discard everything else in the signal. They propose Unified Audio Schema (UAS), a JSON target with three parts, drawing on Laver's layers of speech information:

  • Transcription
  • Paralinguistics: age, gender, emotion, accent, prosody, timbre
  • Non-linguistic Events: a scene description plus discrete and continuous sounds

Instead of manual labeling, a three-stage automatic pipeline converts existing ASR corpora. An audio captioner describes the acoustics, an LLM merges that caption with the reference transcript into the schema, and a validation step filters out hallucinated or inconsistent samples. The resulting UAS-Audio was trained on both continuous and discrete AudioLLM architectures. It improves MMSU perception accuracy by roughly 11 points over same-size state-of-the-art models without degrading reasoning, transcription or speech synthesis. The work suggests that the structure of supervision, not only scale, is key to perception-aware audio models.

Key points

  • Benchmark results: UAS-Audio gains 10.9% (absolute) in perception accuracy on MMSU. It is best on MMAR (60.1%), competitive on MMAU, and has the top average across the three benchmarks (65.2%).
  • Speech generation: On Seed-TTS, UAS-Audio reaches an average WER of 1.6, compared with 1.9 for Qwen2.5-Omni and 2.7 for Step-Audio2-mini.
  • Transcription robustness: Producing the full JSON instead of plain ASR output changes WER by only about 0.1. That is 2.2 vs 2.3 on LibriSpeech test-clean and 2.3 vs 2.4 on AISHELL.
  • Ablations: Removing either the UAS supervision or the UAS-QA strategy consistently lowers performance. Code and model are released at github.com/Tencent/Unified_Audio_Schema.
  • Limitations: Validation covers only English and Chinese, so low-resource languages (such as Ukrainian) and code-switching remain untested. Paralinguistic analysis covers only the primary speaker, so heavily overlapping speech is not addressed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,267 citations🏢TencentShenzhen · China163 works · 21,032 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 519 works

Cite

Cite as Linhao Zhang, Yuhan Song, Aiwei Liu, Chuhan Wu, Sijun Zhang, Wei Jia, Yuan Liu, Houfeng Wang and Zhou Xiao. 2026. Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.967

@inproceedings{zhang2026beyond,
  title = {{Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs}},
  author = {Linhao Zhang, Yuhan Song, Aiwei Liu, Chuhan Wu, Sijun Zhang, Wei Jia, Yuan Liu, Houfeng Wang and Zhou Xiao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.967},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.967}
}

References

51 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, Shengpeng Ji, Yabin Li, Zerui Li, Heng Lu, Haoneng Luo, Xiang Lv, Bin Ma, Ziyang Ma, Chongjia Ni, and 14 others. 2024. FunAudioLLM: Voice Understanding and Generation Foundation Models for Natur… — [arXiv:2407.04051](https://arxiv.org/abs/2407.04051) find
  2. Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, and 27 others. 2024. Seed-TTS: A Family of High-Quality Ve…
  3. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common Voice: A Massively-Multilingual Speech Corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 421…
  4. Jisheng Bai, Haohe Liu, Mou Wang, Dongyuan Shi, Wenwu Wang, Mark D. Plumbley, Woon-Seng Gan, and Jianfeng Chen. 2025. AudioSetCaps: An Enriched Audio-Caption Dataset Using Automated Generation Pipeline With Large Audio and Language Models. IEEE Transactions on Audio, Speech and Language Processing,… find
  5. Evelina Bakhturina, Vitaly Lavrukhin, Boris Ginsburg, and Yang Zhang. 2021. find
  6. Hi-Fi Multi-Speaker English TTS Dataset. In Interspeech 2021, pages 2776–2780. find
  7. Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017. AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline. In 2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COC… find
  8. Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan. 2021. GigaSpeech: An Evolving, Mu… find
  9. Heinrich Dinkel, Gang Li, Jizhong Liu, Jian Luan, Yadong Niu, Xingwei Sun, Tianzi Wang, Qiyang Xiao, Junbo Zhang, and Jiahao Zhou. 2026. MiDashengLM: Efficient Audio Understanding with General Audio Captions. Preprint, arXiv:2508.03983. — [arXiv:2508.03983](https://arxiv.org/abs/2508.03983) find
  10. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. Preprint, arXiv:2410.00037.
  11. Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2025. LLaMA-Omni: Seamless Speech Interaction with Large Language Models. In International Conference on Learning Representations, volume 2025, pages 57607–57624.
  12. Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio Set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pag… find
  13. Sreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. 2025. Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models. In The Thirty-ninth Annual Con… find
  14. Yuan Gong, Hongyin Luo, Alexander Liu, Leonid Karlinsky, and James R Glass. 2024. Listen, Think, and Understand. In International Conference on Learning Representations, volume 2024, pages 18516– 18545. find
  15. Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. 2024. find
  16. Emilia: An Extensive, Multilingual, and Diverse Speech Dataset For Large-Scale Speech Generation. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 885–890. find
  17. Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, and Furu Wei. 2024. WavLLM: Towards Robust and Adaptive Speech Large Language Model. In Findings of the Association for Computational Linguistics: EMNLP 2024,… find
  18. Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, Peng Liu, Ruihang Miao, Wang You, Xi Chen, Xuerui Yang, Yechang Huang, Yuxiang Zhang, Zheng Gong, Zixin Zhang, and 126 others. 2025. Step-Audio: Unified Understanding and Generation i…
  19. KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, and 21 others. 2025. Kimi-Audio Technical Report. Preprint, arXiv:2504.18425.
  20. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. In Advances in Neural Information Processing Systems, volume 33, pages 17022–17033. Curran Associates, Inc.
  21. Yejin Kwon, Taewoo Kang, Hyunsoo Yoon, and Chang-ouk Kim. 2025. M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models. Preprint, arXiv:2510.19358. find
  22. John Laver. 1994. Principles of Phonetics. Cambridge University Press. find
  23. Gang Li, Jizhong Liu, Heinrich Dinkel, Yadong Niu, Junbo Zhang, and Jian Luan. 2025a. Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering. Preprint, arXiv:2503.11197. — [arXiv:2503.11197](https://arxiv.org/abs/2503.11197) find
  24. Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, Jianhua Xu, Haoze Sun, Zenan Zhou, and Weipeng Chen. 2025b. Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction. Preprint, arXiv:2502.17239.
  25. Xinjian Li, Shinnosuke Takamichi, Takaaki Saeki, William Chen, Sayaka Shiota, and Shinji Watanabe. 2023. Yodas: Youtube-Oriented Dataset for Audio and Speech. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8. find
  26. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. 2023. Flow Matching for Generative Modeling. In The Eleventh International Conference on Learning Representations.
  27. Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, Kai Li, Keliang Li, Siyou Li, Xinfeng Li, Xiquan Li, Zheng Lian, Yuzhe Liang, Minghao Liu, Zhikang Niu, and 15 others. 2025. MMAR: A Challenging Benchmark for Deep Reasoning in Spe… find
  28. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. find
  29. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. MLS: A Large-Scale Multilingual Dataset for Speech Research. In Interspeech 2020, pages 2757–2761.
  30. S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2025. MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark. In International Conference on Learning Representations, volume 2025, pages 8492…
  31. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Preprint, arXiv:2402.03300.
  32. Yuhan Song, Linhao Zhang, Chuhan Wu, Aiwei Liu, Wei Jia, Houfeng Wang, and Zhou Xiao. 2026. StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs. In The Fourteenth International Conference on Learning Representations. find
  33. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. 2024. SALMONN: Towards Generic Hearing Abilities for Large Language Models. In International Conference on Learning Representations, volume 2024, pages 16607–16629.
  34. Core Team, Dong Zhang, Gang Wang, Jinlong Xue, Kai Fang, Liang Zhao, Rui Ma, Shuhuai Ren, Shuo Liu, Tao Guo, Weiji Zhuang, Xin Zhang, Xingchen Song, Yihan Yan, Yongzhe He, Cici, Bowen Shen, Chengxuan Zhu, Chong Ma, and 81 others. 2025. MiMo-Audio: Audio Language Models are Few-Shot Learners. Preprin…
  35. Zhou Zhao, Chang Zhou, and Jingren Zhou. 2024. AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1979–1998, Bangkok, Thailand. Association for Com…
  36. Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. 2017. Neural Discrete Representation Learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  37. Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. In Interspeech 2019, pages 1526– 1530.
  38. Dingdong WANG, Junan Li, Jincenzi Wu, Dongchao Yang, Xueyuan Chen, Tianhua Zhang, and Helen M. Meng. 2026. MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark. In The Fourteenth International Conference on Learning Representations. find
  39. Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 2024. GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot. Preprint, arXiv:2412.02612.
  40. Xiong Wang, Yangze Li, Chaoyou Fu, Yike Zhang, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. 2025. Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machin… find
  41. Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng. 2022. WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech… find
  42. Gijs Wijngaard, Elia Formisano, Michele Esposito, and Michel Dumontier. 2025. AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of Sound. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  43. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15757–15773, Singapore. Associa…
  44. Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, Mingrui Chen, Peng Liu, Wang You, Xiangyu Tony Zhang, Xingyuan Li, Xuerui Yang, Yayue Deng, Yechang Huang, Yuxin Li, and 90 others. 2025. Step-Audio 2 Technical Report. Preprint, arXiv:250…
  45. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025a. Qwen2.5-Omni Technical Report. Preprint, arXiv:2503.20215.
  46. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025b. Qwen3-Omni Technical Report. Preprint, arXiv:2509.17765.
  47. Linhao Zhang, Dehong Ma, Xiaodong Zhang, Xiaohui Yan, and Houfeng Wang. 2020. Graph LSTM with Context-Gated Mechanism for Spoken Language Understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9539–9546. find
  48. Linhao Zhang and Houfeng Wang. 2019. Using Bidirectional Transformer-CRF for Spoken Language Understanding. In Natural Language Processing and Chinese Computing, pages 130–141, Cham. Springer International Publishing. find
  49. Linhao Zhang, Jian Zhang, Bokai Lei, Chuhan Wu, Aiwei Liu, Wei Jia, and Xiao Zhou. 2025. WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wild. Preprint, arXiv:2506.21875. — [arXiv:2506.21875](https://arxiv.org/abs/2506.21875) find
  50. Junichi Yamagishi, Christophe Veaux, and Kirsten Mac-Donald. 2019. CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92). University of Edinburgh. The Centre for Speech Technology Research (CSTR). find
  51. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-beyond-transcription-unified-audio-schema/