Elamur Research⌕
☰
home / papers / 2025 / English Pronunciation Evaluation without Complex Joint Training: LoRA Fine-tuned Speech Multimodal LLM

English Pronunciation Evaluation without Complex Joint Training: LoRA Fine-tuned Speech Multimodal LLM

Paper · 2025 · Taekyung Ahn and Hosung Nam

Summary

Pronunciation tutoring software usually splits the work into two problems. Automatic Pronunciation Assessment (APA) gives an utterance scores. Mispronunciation Detection and Diagnosis (MDD) recognises what was actually said, down to the phonemes. Earlier joint systems still needed separate heads, datasets and training stages for each task. The authors take Microsoft's Phi-4-multimodal-instruct, a speech-capable LLM of about 5.8B parameters, and adapt it with LoRA on Speechocean762 so that one model handles both tasks. Task-specific control tokens tell the model which task to perform. The model returns scores or transcripts as JSON. Its predicted scores correlate with human ratings above 0.7 PCC, and both WER and PER stay below 0.15. Training only the LoRA layers worked about as well as also unfreezing the audio encoder and projector. The work shows that a general multimodal LLM, adapted cheaply, can replace a multi-stage CAPT pipeline for English L2 learners. Prosody evaluation remains weak, and inference is heavy.

Key points

  • Model and data: the base is Phi-4-multimodal-instruct, which uses a Mixture-of-LoRAs design. The data is Speechocean762, with 2,500 training and 2,500 test utterances, each split from 125 speakers and roughly 2h40m–2h50m of audio.
  • Task switching: the tokens <|APA|> and <|MDD|> are prepended to the prompts during supervised fine-tuning. The baseline is the untuned model run without these tokens.
  • Output formats: APA covers accuracy, fluency, prosodic and total scores on a 0–10 rubric. MDD produces a word transcript plus phonemes in CMU Pronouncing Dictionary notation, with <unk> marking unclear segments.
  • Results: PCC with human scores exceeds 0.7, and WER and PER are both below 0.15. LoRA-only training is comparable to also unfreezing the audio encoder and projector.
  • Analysis and limits: the authors check whether accuracy scores correlate negatively with MDD phoneme error rate. Prosody and fluency could not be validated well with this dataset. Future work includes quantization for speed and adding lip-movement video.

Interesting ideas

  • I1. One shared model serves two related tasks, selected only by a special token at the start of the input, so no new heads or separate training runs are needed.
  • I2. The prompt spells out a scoring rubric with explicit score bands and demands a fixed JSON shape, which keeps model judgments parseable and comparable across runs.
  • I3. To show that a learned score reflects real quality rather than a memorised data-to-number mapping, the authors test whether it moves inversely with an independent error measure from the other task.
  • I4. Before unfreezing more of a network, the authors ran a controlled ablation and found that adapter-only training already matched the costlier setup.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could take a dialect or scheme token, such as plain UD versus a converted treebank dialect, so one transformer serves several annotation conventions.
  • I2 (learn): LLM teacher tiers could get banded rubrics and a strict JSON schema for their verdicts on silver parses or frames, making disagreements machine-checkable.
  • I3 (learn): as a negative-control style check, parser or frame confidence should correlate with disagreement among the witness parsers, and a missing correlation would flag a meaningless score.
  • I4 (tree): when adapting gparse to new treebanks, adapter-only versus encoder-unfreezing runs should be compared on LAS before paying for full fine-tuning.
  • The output format reserves an explicit <unk> marker for segments the model cannot resolve, instead of forcing a guess. — (answer): readers answering fable QA or school tests could emit an explicit "unknown" or abstain slot, which keeps every answer traceable and honest about gaps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksEvaluation & benchmarks · 1,220 worksVision & language · 657 works

Cite

Cite as Taekyung Ahn and Hosung Nam. 2025. English Pronunciation Evaluation without Complex Joint Training: LoRA Fine-tuned Speech Multimodal LLM. arXiv:2509.02915.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2509.02915 · arXiv 2509.02915

@misc{ahn2025english,
  title = {{English Pronunciation Evaluation without Complex Joint Training: LoRA Fine-tuned Speech Multimodal LLM}},
  author = {Taekyung Ahn and Hosung Nam},
  year = {2025},
  doi = {10.48550/arxiv.2509.02915},
  eprint = {2509.02915},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.02915}
}

References

25 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, et al. 2025. Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs. arXiv preprint arXiv:2503.01743 (2025). ↗
  2. A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli. 2022. XLS-R: Self-supervised cross-lingual speech representation learning at scale. In Proc. Interspeech 2022. 2278–2282. doi:10.21437/Interspeech.2022-143 ↗
  3. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. 2020. Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In Advances in Neural Information Processing Systems, Vol. 33. 12449–12460.
  4. J. Bernstein, M. Cohen, H. Murveit, D. Rtischev, and M. Weintraub. 1990. Automatic evaluation and training in English pronunciation. In Proceedings of the ICSLP-90: 1990 International Conference on Spoken Language Processing. 1185– 1188. find
  5. A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli. 2021. Unsupervised cross-lingual representation learning for speech recognition. In Proc. Interspeech 2021. 2426–2430. doi:10.21437/Interspeech.2021-329 ↗
  6. M. Eskenazi. 2009. An overview of spoken language technology for education. Speech Communication 51, 10 (2009), 832–844. ↗
  7. Y. Feng, G. Fu, Q. Chen, and K. Chen. 2020. SED-MDD: Towards sentence dependent end-to-end mispronunciation detection and diagnosis. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3492–3496. find
  8. J. Fouz-González. 2015. Trends and directions in computer-assisted pronunciation training. In Investigating English pronunciation: Trends and directions. 314–342. ↗
  9. H. Franco, V. Abrash, K. Precoda, H. Bratt, R. Rao, and J. Butzberger. 2000. The SRI EduSpeak system: recognition and pronunciation scoring for language learning. In Proceedings of Intelligent Speech Technology in Language Learning, InSTiLL-2000. find
  10. H. Franco, L. Neumeyer, V. Digalakis, and O. Ronen. 2000. Combination of machine scores for automatic grading of pronunciation quality. Speech Communication 30 (2000), 121–130. ↗
  11. K. Fu, L. Peng, N. Yang, and S. Zhou. 2024. Pronunciation Assessment with Multi-modal Large Language Models. arXiv preprint arXiv:2407.09209 (2024). ↗
  12. Y. Getman, R. Al-Ghezi, K. Voskoboinik, T. Grósz, M. Kurimo, G. Salvi, T. Svendsen, and S. Strömbergsson. 2022. Wav2vec2-Based Speech Rating System for Children with Speech Sound Disorder. In Interspeech 2022. ISCA. find
  13. Y. Gong, Z. Chen, I. H. Chu, P. Chang, and J. Glass. 2022. Transformer-based multi-aspect multi-granularity non-native english speaker pronunciation assessment. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 7262–7266.
  14. J. Heo. 2024. Practical AI application development using LLMs. onlybook. find
  15. Y. Hong. 2021. Automated evaluation of pronunciation for L2 English learners. Ph. D. Dissertation. Graduate School, Korea University. find
  16. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024). ↗
  17. Microsoft. 2025. Phi-4-multimodal-instruct. Hugging Face. https://huggingface.co/microsoft/Phi-4-multimodal-instruct ↗
  18. L. Peng, K. Fu, B. Lin, D. Ke, and J. Zhang. 2021. A Study on Fine-Tuning wav2vec2. 0 Model for the Task of Mispronunciation Detection and Diagnosis. In Interspeech. ISCA. find
  19. X. Qian, H. Meng, and F. Soong. 2010. The use of DBN-HMMs for mispronunciation detection and diagnosis in L2 English to support computer-aided pronunciation training. In Proceedings of INTERSPEECH. 775–778. find
  20. H. Ryu, S. Kim, and M. Chung. 2023. A Joint Model for Pronunciation Assessment and Mispronunciation Detection and Diagnosis with Multi-task Learning. In Proc. Interspeech 2023. 959–963. doi:10.21437/Interspeech.2023-337 ↗
  21. K. Wang, L. He, K. Liu, Y. Deng, W. Wei, and S. Zhao. 2025. Exploring the Potential of Large Multimodal Models as Effective Alternatives for Pronunciation Assessment. arXiv preprint arXiv:2503.11229 (2025). ↗
  22. R. Weide et al. 1998. The Carnegie Mellon pronouncing dictionary (Release 0.6). Retrieved from www.cs.cmu.edu. find
  23. S. M. Witt and S. J. Young. 2000. Phone-level pronunciation scoring and assessment for interactive language learning. Speech Communication 30, 2-3 (2000), 95–108. ↗
  24. M. Yang, K. Hirschi, S. D. Looney, O. Kang, and J. H. L. Hansen. 2022. Improving Mispronunciation Detection with Wav2vec2-Based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessment. In Interspeech 2022. ISCA. find
  25. J. Zhang, Z. Zhang, Y. Wang, Z. Yan, Q. Song, Y. Huang, et al. 2021. speechocean762: An open-source non-native english speech corpus for pronunciation assessment. arXiv preprint arXiv:2104.01378 (2021). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ahn-2025-english-pronunciation-evaluation-complex-joint/