Elamur Research⌕
☰
home / papers / 2025 / QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions

QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions

Paper · 2025 · Siyin Wang, Wenyi Yu, Xianzhao Chen et al.

Summary

Automatic speech quality assessment usually outputs a single number, typically a predicted MOS, which gives no explanation of why a sample sounds good or bad. This paper introduces QualiSpeech, an English dataset in which each clip is annotated across 11 low-level aspects. Seven of these are rated numerically: overall quality, along with naturalness, the effort required to listen, continuity, speaking speed, distortion, and noise. Four receive short textual descriptions: noise type and timing, distortion type and timing, unnatural pauses, and voice traits. Each clip also has a free-text comment that reasons toward the overall rating.

The descriptions were drafted by GPT from human annotations. Annotators then corrected hallucinations and filled in gaps.

The authors also release the QualiSpeech Benchmark, a multiple-choice test of low-level perception in auditory LLMs. On this benchmark, existing open-source models perform poorly. Fine-tuning SALMONN-7B on the data lets it describe noise and distortion, including their type and timing, with useful accuracy. However, its reasoning toward the overall score is held back by its weak Vicuna backbone. The resource matters because it moves quality evaluation toward explainable judgments that cover both synthetic and real speech.

Key points

  • The training split contains 10,558 samples drawn from several sources:

    • BVCC (4,546)
    • NISQA SIM (2,883)
    • NISQA LIVE (529)
    • Gigaspeech (2,000)
    • 600 clips from 10 recent open-source TTS systems, such as ChatTTS, CosyVoice, F5-TTS and XTTS v2

    Overall, the split is roughly half synthetic and half real or simulated-real speech.

  • Including numeric scores inside the generated descriptions improved output quality. Text length had little effect, and removing hallucinations from the training comments proved important.

  • When given ground-truth low-level features, text LLMs predicted the overall score with these accuracies:

    • Vicuna-v1.5-7B: 0.28
    • GPT-4o-mini: 0.46, which beat all fine-tuned auditory LLMs

  • One limitation is that each sample was rated by a single listener.

  • The dataset is available on Hugging Face (tsinghua-ee/QualiSpeech) under CC BY-NC-SA 4.0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Siyin Wang, Wenyi Yu, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Yu Tsao, Junichi Yamagishi, Yuxuan Wang and Chao Zhang. 2025. QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1150

@inproceedings{wang2025qualispeech,
  title = {{QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions}},
  author = {Siyin Wang, Wenyi Yu, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Yu Tsao, Junichi Yamagishi, Yuxuan Wang and Chao Zhang},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1150},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1150}
}

References

56 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chen Chen, Yuchen Hu, Siyin Wang, Helin Wang, Zhehuai Chen, Chao Zhang, Chao-Han Huck Yang, and Eng Siong Chng. 2025. Audio large language models can be descriptive speech quality evaluators. In Proc. ICLR, Singapore. find
  2. Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. 2021. GigaSpeech: An evolving, multi-domain ASR corpus with 10,000 hours of transcribed audio. In Proc. Interspeech, Florence. find
  3. Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. 2023. BEATs: Audio pre-training with acoustic tokenizers. In Proc. ICML, Honolulu. find
  4. Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. 2024a. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885.
  5. Zhehuai Chen, He Huang, Andrei Andrusenko, Oleksii Hrinchuk, Krishna C Puvvada, Jason Li, Subhankar Ghosh, Jagadeesh Balam, and Boris Ginsburg. 2024b. SALM: Speech-augmented language model with in-context learning for speech recognition and translation. In Proc. ICASSP, Seoul. find
  6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. See https://vicuna.lmsys. org (accessed 14 April 2023), 2(3):6. ↗
  7. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-Audio technical report. arXiv preprint arXiv:2407.10759.
  8. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919.
  9. Erica Cooper, Wen-Chin Huang, Tomoki Toda, and Junichi Yamagishi. 2022. Generalization ability of MOS prediction networks. In Proc. ICASSP, Singapore. find
  10. Erica Cooper, Wen-Chin Huang, Yu Tsao, Hsin-Min Wang, Tomoki Toda, and Junichi Yamagishi. 2023. The VoiceMOS challenge 2023: Zero-shot subjective speech quality prediction for multiple domains. In Proc. ASRU, Taipei. find
  11. Erica Cooper and Junichi Yamagishi. 2021. How do voices from past speech synthesis challenges compare today? In Proc. SSW, Budapest. find
  12. Soham Deshmukh, Shuo Han, Hazim Bukhari, Benjamin Elizalde, Hannes Gamper, Rita Singh, and Bhiksha Raj. 2024. Audio Entailment: Assessing deductive reasoning for audio understanding. arXiv preprint arXiv:2407.18062. — [arXiv:2407.18062](https://arxiv.org/abs/2407.18062) find
  13. Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. 2024. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407. find
  14. Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cutler, Hannes Gamper, Mehrsa Golestaneh, and Robert Aichner. 2023. find
  15. ICASSP 2023 Deep Noise Suppression Challenge. In Proc. ICASSP, Rhodes Island. find
  16. Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et al. 2024. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts. In Proc. SLT, Macao. find
  17. Wei Kang, Xiaoyu Yang, Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, and Daniel Povey. 2024. Libriheavy: A 50,000 hours ASR corpus with punctuation casing and context. In Proc. ICASSP, Seoul. find
  18. Simon King and Vasilis Karaiskos. 2016. find
  19. The Blizzard Challenge 2016. In The Blizzard Challenge. find
  20. Chen-Chou Lo, Szu-Wei Fu, Wen-Chin Huang, Xin Wang, Junichi Yamagishi, Yu Tsao, and Hsin-Min Wang. 2019. MOSNet: Deep learning based objective assessment for voice conversion. In Proc. Interspeech, Graz. find
  21. Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, He Huang, Boris Ginsburg, Yu-Chiang Frank Wang, and Hung-yi Lee. 2024. DeSTA: Enhancing speech language models through descriptive speech-text alignment. In Proc. Interspeech, Kos Island. find
  22. Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, et al. 2024. Prompting large language models with speech recognition abilities. In Proc. ICASSP, Seoul. find
  23. Georgia Maniati, Alexandra Vioni, Nikolaos Ellinas, Karolos Nikitaras, Konstantinos Klapsas, June Sig Sung, Gunu Jho, Aimilios Chalamandaris, and Pirros Tsiakoulis. 2022. SOMOS: The Samsung open MOS dataset for the evaluation of neural text-to-speech synthesis. In Proc. Interspeech, Incheon. find
  24. Eduardo Fonseca, Jordi Pons Puig, Xavier Favory, Frederic Font Corbera, Dmitry Bogdanov, Andres Ferraro, Sergio Oramas, Alastair Porter, and Xavier Serra. 2017. Freesound datasets: A platform for the creation of open audio datasets. In Proc. ISMIR, Suzhou. find
  25. Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller. 2021. NISQA: A deep CNN-self-attention model for multidimensional speech quality prediction with crowdsourced datasets. In Proc. Interspeech, Brno. find
  26. Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In Proc. ICASSP, New Orleans. find
  27. Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, and Furu Wei. 2024. WavLLM: Towards robust and adaptive speech large language model. In Findings of EMNLP, Miami. find
  28. Yi Hu and Philipos C Loizou. 2007. Evaluation of objective quality measures for speech enhancement. IEEE Transactions on Audio, Speech, and Language Processing, 16(1):229–238. find
  29. Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang, Andy T Liu, Chen-An Li, Yu-Xiang Lin, Wei-Cheng Tseng, Anuj Diwan, Yi-Jen Shih, Jiatong Shi, et al. 2024. Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks. arXiv preprint… — [arXiv:2411.05361](https://arxiv.org/abs/2411.05361) find
  30. Zeyu Jin, Jia Jia, Qixin Wang, Kehan Li, Shuoyi Zhou, Songtao Zhou, Xiaoyu Qin, and Zhiyong Wu. 2024. SpeechCraft: A fine-grained expressive speech dataset with natural language description. In Proc. ACM MM, Melbourne. find
  31. Brian Patton, Yannis Agiomyrgiannakis, Michael Terry, Kevin Wilson, Rif A Saurous, and D Sculley. 2016. AutoMOS: Learning a non-intrusive assessor of naturalness-of-speech. In Proc. NIPS End-to-end Learning for Speech and Audio Processing Workshop, Barcelona. find
  32. Puyuan Peng, Po-Yao Huang, Shang-Wen Li, Abdelrahman Mohamed, and David Harwath. 2024. Voice-Craft: Zero-shot speech editing and text-to-speech in the wild. In Proc. ACL, Bangkok. find
  33. Zengyi Qin, Wenliang Zhao, Xumin Yu, and Xin Sun. 2023. Openvoice: Versatile instant voice cloning. arXiv preprint arXiv:2312.01479. — [arXiv:2312.01479](https://arxiv.org/abs/2312.01479) find
  34. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proc. ICML, Honolulu.
  35. Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019. Fastspeech: Fast, robust and controllable text to speech. In Proc. NIPS, Vancouver. find
  36. Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. 2023. AudioPaLM: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925.
  37. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. UTMOS: UTokyo-SaruLab system for VoiceMOS Challenge 2022. In Proc. Interspeech, Incheon. find
  38. Yaoxun Xu, Hangting Chen, Jianwei Yu, Qiaochu Huang, Zhiyong Wu, Shi-Xiong Zhang, Guangzhi Li, Yi Luo, and Rongzhi Gu. 2024. Secap: Speech emotion captioning with large language model. In AAAI, Vancouver. find
  39. Oscar Figuerola Salas, Velibor Adzic, and Hari Kalva. 2013. Subjective quality evaluations using crowdsourcing. In Proc. PCS, San Jose. find
  40. Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. 2024. AIR-Bench: Benchmarking large audio-language models via generative comprehension. In Proc. ACL, Bangkok.
  41. Thibault Sellam, Ankur Bapna, Joshua Camp, Diana Mackinnon, Ankur P Parikh, and Jason Riesa. 2023. SQuId: Measuring speech naturalness in many languages. In Proc. ICASSP, Rhodes Island. find
  42. Suwon Shon, Kwangyoun Kim, Yi-Te Hsu, Prashant Sridhar, Shinji Watanabe, and Karen Livescu. 2024. DiscreteSLU: A large language model with self-supervised discrete speech units for spoken language understanding. arXiv preprint arXiv:2406.09345. — [arXiv:2406.09345](https://arxiv.org/abs/2406.09345) find
  43. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024a. SALMONN: Towards generic hearing abilities for large language models. In Proc. ICLR, Vienna.
  44. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Jun Zhang, Lu Lu, Zejun Ma, Yuxuan Wang, et al. 2024b. Can large language models understand spatial audio? arXiv preprint arXiv:2406.07914. — [arXiv:2406.07914](https://arxiv.org/abs/2406.07914) find
  45. Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. 2013. DEMAND: A collection of multi-channel recordings of acoustic noise in diverse environments. In Poc. ICAASA, Montreal. find
  46. Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F Chen. 2024a. AudioBench: A universal benchmark for audio large language models. arXiv preprint arXiv:2406.16020.
  47. Zhao Yi, Wen-Chin Huang, Xiaohai Tian, Junichi Yamagishi, Rohan Kumar Das, Tomi Kinnunen, Zhen-Hua Ling, and Tomoki Toda. 2020. find
  48. Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion. In Proc. VCCBC, Shanghai. find
  49. Zhiyuan You, Zheyuan Li, Jinjin Gu, Zhenfei Yin, Tianfan Xue, and Chao Dong. 2025. Depicting beyond scores: Advancing image quality assessment through multi-modal language models. In ECCV, London. find
  50. Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. Connecting speech encoder and large language model for ASR. In Proc. ICASSP, Seoul. find
  51. Ryandhimas E Zezario, Sabato M Siniscalchi, Hsin-Min Wang, and Yu Tsao. 2024. A study on zero-shot non-intrusive speech assessment using large language models. arXiv preprint arXiv:2409.09914. — [arXiv:2409.09914](https://arxiv.org/abs/2409.09914) find
  52. Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath. 2024. BAT: Learning to reason about spatial sounds with large language models. In Proc. ICML, Vienna. find
  53. Mingqiu Wang, Wei Han, Izhak Shafran, Zelin Wu, Chung-Cheng Chiu, Yuan Cao, Nanxin Chen, Yu Zhang, Hagen Soltau, Paul K Rubenstein, et al. 2023. SLM: Bridge the thin gap between speech and text foundation models. In ASRU, Taipei.
  54. Siyin Wang, Wenyi Yu, Yudong Yang, Changli Tang, Yixuan Li, Jimin Zhuang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, et al. 2024b. Enabling auditory large language models for automatic speech quality evaluation. arXiv preprint arXiv:2409.16644. — [arXiv:2409.16644](https://arxiv.org/abs/2409.16644) find
  55. Matthew B Winn and Katherine H Teece. 2021. Listening effort is not the same as speech intelligibility score. Trends in Hearing, 25. find
  56. Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, et al. 2023. On decoder-only architecture for speech-to-text and large language model integration. In Proc. ASRU, Teipei. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2025-qualispeech-speech-quality-assessment-dataset/