Elamur Research⌕
☰
home / papers / 2026 / SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation

SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation

Paper · 2026 · Hui Wang, Jinghua Zhao, Yifan Yang et al.

Summary

Judging how good synthetic speech sounds is still difficult. Common tools such as MOS ratings, A/B tests, or MCD produce a single number or a binary choice, which explains nothing about the judgment. Learned predictors such as UTMOS are usually limited to one task and one language. The authors propose SpeechLLM-as-Judges, a paradigm where a speech-capable LLM gives structured, explained verdicts on speech quality. To support it, they built SpeechEval, a multilingual, human-annotated dataset covering English, Chinese, Japanese and French. It spans four tasks: descriptive quality assessment, pairwise comparison, improvement suggestions, and deepfake detection. On top of this data they trained SQ-LLM in two stages, first chain-of-thought instruction tuning and then reward optimization with GRPO. SQ-LLM beats untuned audio LLMs, cascaded LLM pipelines, and fine-tuned baselines across the tasks, and it stays consistent across languages. The work matters because it points toward one interpretable, multi-task evaluator for TTS and other speech generation systems, replacing several narrow metrics. Code, models and data are released publicly.

Key points

  • SpeechEval scale: 32,207 unique utterances and 128,754 annotations in four languages, with natural-language human annotations rather than scalar scores.
  • Tasks: SQA (assessment), SQC (comparison), SQI (improvement suggestion) and DSD (deepfake detection).
  • Deepfake detection: SQ-LLM reaches 6.249% EER, 0.142 minDCF and 89.358% accuracy. This beats expert systems (RawNet2, AASIST, AASIST2) and a fine-tuned Qwen2-Audio-7B (8.593% EER).
  • Ablation: Moving from vanilla fine-tuning to CoT to CoT+GRPO improves every task. The largest gain is on SQI, where the LLM score rises from 6.733 to 7.018 to 7.420.
  • Multilingual results and limitations: LLM scores on the generation-style tasks are generally above 6.5 in all four languages. Detection is strongest for Japanese and weaker for Chinese and French. The authors note that coverage is limited to four languages and a fixed task set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓Nankai UniversityTianjin · China24 works · 762 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

Cite

Cite as Hui Wang, Jinghua Zhao, Yifan Yang, Shujie Liu, Junyang Chen, Yanzhe Zhang, Shiwan Zhao, Jinyu Li, Jiaming Zhou, Haoqin Sun, Yan Lu and Yong Qin. 2026. SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.349

@inproceedings{wang2026speechllm,
  title = {{SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation}},
  author = {Hui Wang, Jinghua Zhao, Yifan Yang, Shujie Liu, Junyang Chen, Yanzhe Zhang, Shiwan Zhao, Jinyu Li, Jiaming Zhou, Haoqin Sun, Yan Lu and Yong Qin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.349},
  url = {https://doi.org/10.18653/v1/2026.acl-long.349}
}

References

55 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4211–4215.
  2. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick Von Platen, Yatharth Saraf, Juan Pino, and 1 others. 2021. Xls-r: Self-supervised cross-lingual arXiv Chen Chen, Yuchen Hu, Siyin Wang, Helin Wang, Zhehuai Chen, Chao Zhang, Chao-Han Huck Yang… — [arXiv:2501.17202](https://arxiv.org/abs/2501.17202) find
  3. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, and 1 others. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518. find
  4. Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, and 1 others. 2025b. Neural codec language models are zero-shot text to speech synthesizers. IEEE Transactions on Audio, Speech and Language Processing.
  5. Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. 2024. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. ArXiv, abs/2410.06885.
  6. Xize Cheng, Dongjie Fu, Xiaoda Yang, Minghui Fang, Ruofan Hu, Jingyu Lu, Bai Jionghao, Zehan Wang, Shengpeng Ji, Rongjie Huang, and 1 others. 2025. Omnichat: Enhancing spoken dialogue systems with scalable synthetic data for diverse scenarios. arXiv preprint arXiv:2501.01384. — [arXiv:2501.01384](https://arxiv.org/abs/2501.01384) find
  7. Chung-Ming Chien, Jheng-Hao Lin, Chien-yu Huang, Po-chun Hsu, and Hung-yi Lee. 2021. Investigating on incorporating pretrained and learnable speaker representations for multi-speaker multi-style text-to-speech. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Proce… find
  8. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. Qwen2-audio technical report. arXiv preprint arXiv: 2407.10759.
  9. Erica Cooper and Junichi Yamagishi. 2021. How do voices from past speech synthesis challenges compare today? arXiv preprint arXiv:2105.02373. — [arXiv:2105.02373](https://arxiv.org/abs/2105.02373) find
  10. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. with expert ratings for automatic text-to-music evaluation. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages…
  11. Heinrich Dinkel, Gang Li, Jizhong Liu, Jian Luan, Yadong Niu, Xingwei Sun, Tianzi Wang, Qiyang Xiao, Junbo Zhang, and Jiahao Zhou. 2025. Midashenglm: Efficient audio understanding with general audio captions. arXiv preprint arXiv: 2508.03983. — [arXiv:2508.03983](https://arxiv.org/abs/2508.03983) find
  12. Chen-Chou Lo, Szu-Wei Fu, Wen-Chin Huang, Xin Wang, Junichi Yamagishi, Yu Tsao, and Hsin-Min Wang. 2019. Mosnet: Deep learning-based objective assessment for voice conversion. In Interspeech 2019, pages 1541–1545. find
  13. Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, and 1 others. 2024. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407. find
  14. Jaime Lorenzo-Trueba, Junichi Yamagishi, Tomoki Toda, Daisuke Saito, Fernando Villavicencio, Tomi Kinnunen, and Zhenhua Ling. 2018. The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods. arXiv preprint arXiv:1804.04262. — [arXiv:1804.04262](https://arxiv.org/abs/1804.04262) find
  15. Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yu Zhang, Ming-Hsuan Yang, and Philip Torr. 2019. Res2net: A new multi-scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence, 43(2):652–662. find
  16. Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller. 2021. Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets. arXiv preprint arXiv:2104.09494. find
  17. Hao Gu, Jiangyan Yi, Chenglong Wang, Jianhua Tao, Zheng Lian, Jiayi He, Yong Ren, Yujie Chen, and Zhengqi Wen. 2025. Allm4add: Unlocking the capabilities of audio large language models for audio deepfake detection. arXiv preprint arXiv:2505.11079. — [arXiv:2505.11079](https://arxiv.org/abs/2505.11079) find
  18. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  19. Pierre-Edouard Honnet, Alexandros Lazaridis, Philip N Garner, and Junichi Yamagishi. 2017. The siwis french speech synthesis database? design and recording of a high quality french database for speech synthesis. Online Database. find
  20. Jee-weon Jung, Hee-Soo Heo, Hemlata Tak, Hye-jin Shim, Joon Son Chung, Bong-Jin Lee, Ha-Jin Yu, and Nicholas Evans. 2022. Aasist: Audio anti-spoofing using integrated spectro-temporal graph attention networks. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processi… find
  21. Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages 5530–5540. PMLR. find
  22. Robert Kubichek. 1993. Mel-cepstral distance measure for objective speech quality assessment. In Proceedings of IEEE pacific rim conference on communications computers and signal processing, volume 1, pages 125–128. IEEE. find
  23. Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. 2019. Neural speech synthesis with transformer network. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6706–6713. find
  24. Cheng Liu, Hui Wang, Jinghua Zhao, Shiwan Zhao, Hui Bu, Xin Xu, Jiaming Zhou, Haoqin Sun, and Yong Qin. 2025. Musiceval: A generative music dataset Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervi… find
  25. Mirco Ravanelli and Yoshua Bengio. 2018. Speaker recognition from raw waveform with sincnet. In 2018 IEEE spoken language technology workshop (SLT), pages 1021–1028. IEEE. find
  26. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022. In Interspeech 2022, pages 4521–4525. find
  27. Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. 2021. Aishell-3: A multi-speaker mandarin tts corpus. In Interspeech 2021, pages 2756–2760. find
  28. Ryosuke Sonobe, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2017. Jsut corpus: free large-scale japanese speech corpus for end-to-end speech synthesis. arXiv preprint arXiv:1711.00354. — [arXiv:1711.00354](https://arxiv.org/abs/1711.00354) find
  29. Suno. 2024. Suno official website. find
  30. Hemlata Tak, Jose Patino, Massimiliano Todisco, Andreas Nautsch, Nicholas Evans, and Anthony Larcher. 2021. End-to-end anti-spoofing with rawnet2. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6369–6373. IEEE. find
  31. Shinnosuke Takamichi, Kentaro Mitsui, Yuki Saito, Tomoki Koriyama, Naoko Tanji, and Hiroshi Saruwatari. 2019. Jvs corpus: free japanese multi-speaker voice corpus. arXiv preprint arXiv:1908.06248. — [arXiv:1908.06248](https://arxiv.org/abs/1908.06248) find
  32. Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, and 1 others. 2025. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. arXiv preprint arXiv:2502.05139. — [arXiv:2502.05139](https://arxiv.org/abs/2502.05139) find
  33. Ashvala Vinay and Alexander Lerch. 2022. Evaluating generative audio systems and their metrics. arXiv preprint arXiv:2209.00130. — [arXiv:2209.00130](https://arxiv.org/abs/2209.00130) find
  34. Hui Wang, Shujie Liu, Lingwei Meng, Jinyu Li, Yifan Yang, Shiwan Zhao, Haiyang Sun, Yanqing Liu, Haoqin Sun, Jiaming Zhou, and 1 others. 2025a. Felle: Autoregressive speech synthesis with token-wise coarse-to-fine flow matching. arXiv preprint arXiv:2502.11128. — [arXiv:2502.11128](https://arxiv.org/abs/2502.11128) find
  35. Hui Wang, Shiwan Zhao, Xiguang Zheng, and Yong Qin. 2023. Ramp: Retrieval-augmented mos prediction via confidence-based dynamic weighting. In Interspeech 2023, pages 1095–1099. find
  36. Hui Wang, Shiwan Zhao, Xiguang Zheng, Jiaming Zhou, Xuechen Wang, and Yong Qin. 2025b. Ramp+: Retrieval-augmented mos prediction with prior knowledge integration. IEEE Transactions on Audio, Speech and Language Processing, 33:1520– 1534. find
  37. Hui Wang, Shiwan Zhao, Jiaming Zhou, Xiguang Zheng, Haoqin Sun, Xuechen Wang, and Yong Qin. 2024. find
  38. Uncertainty-Aware Mean Opinion Score Prediction. In Interspeech 2024, pages 1215–1219. find
  39. Siyin Wang, Wenyi Yu, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Yu Tsao, Junichi Yamagishi, Yuxuan Wang, and Chao Zhang. 2025c. QualiSpeech: A speech quality assessment dataset with natural language reasoning and descriptions. In Proceedings of the 63rd Annual Meeting of the Association for Com…
  40. Siyin Wang, Wenyi Yu, Yudong Yang, Changli Tang, Yixuan Li, Jimin Zhuang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, and Chao Zhang. 2025d. Enabling auditory large language models for automatic speech quality evaluation. In ICASSP 2025 - 2025 IEEE International Conference on Acoust… find
  41. Xin Wang, Junichi Yamagishi, Massimiliano Todisco, Héctor Delgado, Andreas Nautsch, Nicholas Evans, Md Sahidullah, Ville Vestman, Tomi Kinnunen, Kong Aik Lee, and 1 others. 2020. find
  42. Asvspoof 2019: A large-scale public database of synthesized, converted and replayed speech. Computer Speech & Language, 64:101114. find
  43. Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. 2025e. MaskGCT: Zero-shot text-to-speech with masked generative codec transformer. In The Thirteenth International Conference on Learning Representations. find
  44. Yuxuan Wang, R.J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous. 2017. Tacotron: Towards end-to-end speech synthesis. In Interspeech 2017, pages 4006–4010. find
  45. Zhizheng Wu, Zhihang Xie, and Simon King. 2019. The blizzard challenge 2019. In Proc. Blizzard Challenge Workshop, volume 2019, pages 1–24. find
  46. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen2.5-omni technical report. arXiv preprint arXiv: 2503.20215.
  47. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  48. An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, and 1 others. 2025b. Qwen2. 5-1m technical report. arXiv preprint arXiv:2501.15383. — Qwen2.5-1M Technical Report
  49. Yifan Yang, Hui Wang, Bing Han, Shujie Liu, Jinyu Li, Yong Qin, and Xie Chen. 2025c. Towards responsible evaluation for text-to-speech. arXiv preprint arXiv:2510.06927. — [arXiv:2510.06927](https://arxiv.org/abs/2510.06927) find
  50. Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, and 1 others. 2024. Anygpt: Unified multimodal llm with discrete sequence modeling. arXiv preprint arXiv:2402.12226.
  51. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000.
  52. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
  53. Xueyao Zhang, Chaoren Wang, Huan Liao, Ziniu Li, Yuancheng Wang, Li Wang, Dongya Jia, Yuanzhe Chen, Xiulin Li, Zhuo Chen, and 1 others. 2025. Speechjudge: Towards human-level judgment for speech naturalness. arXiv preprint arXiv:2511.07931. — [arXiv:2511.07931](https://arxiv.org/abs/2511.07931) find
  54. Yuxiang Zhang, Jingze Lu, Zengqiang Shang, Wenchao Wang, and Pengyuan Zhang. 2024. Improving short utterance anti-spoofing with aasist2. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11636–11640. IEEE. find
  55. Neutral Happiness Surprise Anger Disgust Fear 1.27% 0.30% 8.65% 91.35% 0.26% 3.64% (a) Emotion distribution: overall on the left, and the breakdown of non-neutral emotions on the right. *Figure 9: Categorical metadata statistics in the SpeechEval assessment data: (a) Emotion distribution, with non-n… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-speechllm-as-judges-general-interpretable/