Elamur Research⌕
☰
home / papers / 2026 / Rubric-Guided Fine-tuning of SpeechLLMs for Multi-Aspect, Multi-Rater L2 Reading-Speech Assessment

Rubric-Guided Fine-tuning of SpeechLLMs for Multi-Aspect, Multi-Rater L2 Reading-Speech Assessment

Summary

Automatically scoring L2 learners reading aloud is difficult because human raters often disagree, and instruction-tuned SpeechLLMs tend to give inflated, poorly calibrated scores. The authors adapt Qwen2-Audio-7B-Instruct to rate English sentences read by Mandarin-speaking children and adults on three dimensions: accuracy, fluency and prosody. Each recording is paired with textual descriptions of the scoring criteria, and a lightweight scoring head is trained using LoRA.

They compare five setups of increasing complexity:

  • plain five-level classification;
  • single-aspect regression;
  • joint regression over all three aspects;
  • joint regression with a Gaussian negative log-likelihood objective that estimates uncertainty;
  • that same model supervised by every individual rater and combined with conformal prediction, which yields calibrated confidence intervals.

All experiments use the public SpeechOcean762 corpus. The fullest variant aligns best with human judgements and outperforms both regression and classification baselines. Fluency and prosody are predicted reliably, while accuracy remains the hardest aspect. The work points toward assessment tools that report a score together with an honest measure of their confidence.

Key points

  • Base model and fine-tuning: Qwen2-Audio-7B-Instruct with LoRA on all linear layers (r = 32, α = 32, dropout 0.1, rsLoRA enabled), giving roughly 10 million trainable parameters while the base weights stay frozen.
  • Data: SpeechOcean762 contains 5,000 English read-speech utterances by Mandarin L1 speakers, split evenly into 2,500 for training and 2,500 for testing. Each utterance is scored by five experts on a 1–10 scale. The Completeness aspect was excluded because its rubric was unclear for this setup.
  • Compared strategies: DiCl, SRR.M, MRR.M, MRR.G and MRR.GC. The authors present the last two, which add Gaussian uncertainty and multi-rater conformal calibration, as novel for automated L2 assessment.
  • Main result: MRR.GC achieved the strongest agreement with human raters. Accuracy was the least reliably assessed dimension.
  • Motivation: Earlier work found a "niceness bias" in zero-shot SpeechLLM scoring. Prior fine-tuning studies also ignored rater variability and predictive uncertainty, which this framework explicitly models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4dgvijh3226x

@inproceedings{parikh2026rubric,
  title = {{Rubric-Guided Fine-tuning of SpeechLLMs for Multi-Aspect, Multi-Rater L2 Reading-Speech Assessment}},
  author = {Parikh, Aditya Kamlesh and Tejedor-García, Cristian and Cucchiarini, Catia and Strik, Helmer},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4dgvijh3226x},
  url = {https://doi.org/10.63317/4dgvijh3226x}
}

References

44 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anastasios N Angelopoulos and Stephen Bates. 2021. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511. find
  2. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460. find our card
  3. Sacha Braun, Eugène Berta, Michael I. Jordan, and Francis Bach. 2025. Multivariate Conformal Prediction via Conformalized Gaussian Scoring. arXiv preprint arXiv:2507.20941. — [arXiv:2507.20941](https://arxiv.org/abs/2507.20941) find
  4. Xinwei Cao, Zijian Fan, Torbjørn Svendsen, and Giampiero Salvi. 2024. A Framework for Phoneme-Level Pronunciation Assessment Using CTC. In Interspeech 2024, pages 302–306. find
  5. Ya-Ning Chang, JSH Taylor, Kathleen Rastle, and Padraic Monaghan. 2020. The relationships between oral language and reading instruction: Evidence from a computational model of reading. Cognitive Psychology, 123:101336. find
  6. Lei Chen, Klaus Zechner, Su-Youn Yoon, Keelan Evanini, Xinhao Wang, Anastassia Loukina, Jidong Tao, Lawrence Davis, Chong Min Lee, Min Ma, et al. 2018. Automated scoring of nonnative speech using the speechrater sm v. 5.0 engine. ETS Research Report Series, 2018(1):1–31. find
  7. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759. find our card
  8. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919. find our card
  9. Catia Cucchiarini, Helmer Strik, and Lou Boves. 2000a. Different aspects of expert pronunciation quality ratings and their relation to scores produced by speech recognition algorithms. Speech Communication, 30(2-3):109–119. find
  10. Catia Cucchiarini, Helmer Strik, and Lou Boves. 2000b. Quantitative assessment of second language learners’ fluency by means of automatic speech recognition technology. The Journal of the Acoustical Society of America, 107(2):989– 999. find
  11. Catia Cucchiarini, Helmer Strik, and Lou Boves. 2002. Quantitative assessment of second language learners’ fluency: Comparisons between read and spontaneous speech. the Journal of the Acoustical Society of America, 111(6):2862– 2873. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,… find
  13. Heejin Do, Yunsu Kim, and Gary Geunbae Lee. 2023. Score-balanced loss for multi-aspect pronunciation assessment. In Interspeech 2023, pages 4998–5002. find
  14. Keelan Evanini, Matthew Mulholland, Rutuja Ubale, Yao Qian, Robert A Pugh, Vikram Ramanarayanan, and Aoife Cahill. 2018. Improvements to an Automated Content Scoring System for Spoken CALL Responses: the ETS Submission to the Second Spoken CALL Shared Task. In Interspeech 2018, pages 2379–2383. find
  15. Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro. 2025. Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities. In Proceedings of the 42nd International Conference on Machine L… find
  16. Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024. GAMA: A large audio-language model with advanced audio understanding and complex reasoning abilities. In Proceedings of the 2024 Conference on Empiric… find
  17. Karianne Megard Grønli, Bente Rigmor Walgermo, Erin M McTigue, and Per Henning Uppstad. 2024. Teachers’ feedback on oral reading: A critical review of its effects and the use of theory in research. Educational Psychology Review, 36(4):121. find
  18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. find
  19. Shin’ichiro Ishikawa. 2023. Effects of raters’ l1, assessment experience, and teaching experience on their assessment of l2 english speech: A study based on the icnale global rating archives. LEARN Journal: Language Education and Acquisition Research Network, 16(2):411–428. find
  20. Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30. find our card
  21. Yoon Kim, Horacio Franco, and Leonardo Neumeyer. 1997. Automatic pronunciation scoring of specific phone segments for language instruction. In Eurospeech, pages 645–648. find
  22. Melanie R Kuhn and Steven A Stahl. 2003. Fluency: A review of developmental and remedial practices. Journal of educational psychology, 95(1):3. find
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pre-training approach. find
  24. Rao Ma, Mengjie Qian, Siyuan Tang, Stefano Bannò, Kate M. Knill, and Mark J.F. Gales. 2025. Assessment of L2 Oral Proficiency using Speech Large Language Models. In Interspeech 2025, pages 5078–5082. find
  25. Reeta Neittaanmäki and Iasonas Lamprianou. 2024. Communal factors in rater severity and consistency over time in high-stakes oral assessment. Language Testing, 41(3):584–605. find
  26. Kirsten W Newell, Robin S Codding, and Tara W Fortune. 2020. Oral reading fluency as a screening tool with english learners: A systematic review. Psychology in the Schools, 57(8):1208– 1239. find
  27. Aditya Kamlesh Parikh, Cristian Tejedor-Garcia, Catia Cucchiarini, and Helmer Strik. 2025a. Enhancing GOP in CTC-Based Mispronunciation Detection with Phonological Knowledge. In Interspeech 2025, pages 5068–5072. find
  28. Aditya Kamlesh Parikh, Cristian Tejedor-Garcia, Catia Cucchiarini, and Helmer Strik. 2025b. Evaluating Logit-Based GOP Scores for Mispronunciation Detection. In Interspeech 2025, pages 2405–2409. find
  29. Aditya Kamlesh Parikh, Cristian Tejedor-Garcia, Catia Cucchiarini, and Helmer Strik. 2025c. Zero-Shot Speech LLMs for Multi-Aspect Evaluation of L2 Speech: Challenges and Opportunities. find
  30. Nhan Phan, Mikko Kuronen, Maria Kautonen, Riikka Ullakonoja, Anna von Zansen, Yaroslav Getman, Ekaterina Voskoboinik, Tamás Grósz, and Mikko Kurimo. 2025. Mispronunciation Detection Without L2 Pronunciation Dataset in Low-Resource Setting: A Case Study in Finland Swedish. In Interspeech 2025, pages… find
  31. Paula J Schwanenflugel, Anne Marie Hamilton, Melanie R Kuhn, Joseph M Wisenbaker, and Steven A Stahl. 2004. Becoming a fluent reader: reading skill and prosodic features in the oral reading of young readers. Journal of educational psychology, 96(1):119. find
  32. Mike Sleeman, John Everatt, Alison Arrow, and Amanda Denston. 2022. The identification and classification of struggling readers based on the simple view of reading. Dyslexia, 28(3):256–275. find
  33. Grant S Smith and David D Paige. 2019. A study of reliability across multiple raters when using the naep and mdfs rubrics to measure oral reading fluency. Reading Psychology, 40(1):34–69. find
  34. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289. find our card
  35. Max van der Velde, Bo Molenaar, Bernard P Veldkamp, Remco Feskens, and Jos Keuning. 2024. What do they say? assessment of oral reading fluency in early primary school children: A scoping review. International Journal of Educational Research, 128:102444. find
  36. Kuo Wang, Xin Qiao, George Sammit, Eric C Larson, Joseph Nese, and Akihito Kamata. 2024. Improving automated scoring of prosody in oral reading fluency using deep learning algorithm. In Frontiers in Education, volume 9, page 1440760. Frontiers Media SA. find
  37. Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. 2023. Viola: Unified codec language models for speech recognition, synthesis, and translation. arXiv preprint arXiv:2305.16107. — [arXiv:2305.16107](https://arxiv.org/abs/2305.16107) find
  38. Silke Maren Witt. 2000. Use of speech recognition in computer-assisted language learning. Ph.D. thesis, University of Cambridge. find
  39. Silke Maren Witt and Steve J Young. 2000. Phone-level pronunciation scoring and assessment for interactive language learning. Speech communication, 30(2-3):95–108. find
  40. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… find our card
  41. Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, et al. 2023. On decoder-only architecture for speech-to-text and large language model integration. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8.… find
  42. Xiaoming Xi, Derrick Higgins, Klaus Zechner, and David Williamson. 2012. A comparison of two scoring methods for an automated speech scoring system. Language Testing, 29(3):371–394. find
  43. Klaus Zechner, Lei Chen, Larry Davis, Keelan Evanini, Chong Min Lee, Chee Wee Leong, Xinhao Wang, and Su-Youn Yoon. 2015. Automated scoring of speaking tasks in the test of english-for-teaching (teft™). ETS Research Report Series, 2015(2):1–17. 8. find
  44. Language Resource References Junbo Zhang and Zhiwen Zhang and Yongqing Wang and Zhiyong Yan and Qiong Song and Yukai Huang and Ke Li and Daniel Povey and Yujun Wang. 2021. speechocean762: An Open-Source Non-Native English Speech Corpus for Pronunciation Assessment. ISCA. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/parikh-2026-rubric-guided-fine-tuning-speechllms/