Elamur Research⌕
☰
home / papers / 2026 / Unlocking Large Audio-Language Models for Interactive Language Learning

Unlocking Large Audio-Language Models for Interactive Language Learning

Paper · 2026 · Hongfu Liu, Zhouying Cui, Xiangming Gu et al.

Summary

Traditional Computer-Assisted Pronunciation Training (CAPT) tools mostly return error locations or numeric scores, which L2 learners often struggle to turn into concrete practice. The authors frame "chat-based pronunciation training": a learner reads a given sentence aloud, and the system replies in text, explaining which words were mispronounced and how to fix them. Because no data existed for this task, they built L2-Arctic-plus on top of L2-Arctic. GPT-4o drafted word-level explanation/suggestion pairs from the existing phoneme-level error labels, and human annotators checked these drafts and had faulty ones regenerated. They then benchmarked ASR+LLM cascades and off-the-shelf audio-language models (ALMs). Cascades performed poorly because the ASR step tends to "fix" mispronunciations, so the LLM never sees them. Existing ALMs also struggled. Instruction-tuning ALMs (audio encoder, projector and LLM) on L2-Arctic-plus in two stages gave the best results under objective metrics, GPT-4o-as-judge evaluation, and human ratings. The work shows that ALMs can serve as more interpretable pronunciation tutors.

Key points

  • L2-Arctic-plus extends L2-Arctic, which covers 24 non-native speakers (12 male, 12 female) with Hindi, Korean, Mandarin, Spanish, Arabic and Vietnamese L1s. It uses the standard 900-sample evaluation split, with error types labelled as Substitution, Deletion or Insertion. Its responses were generated by GPT-4o and verified by three human annotators.
  • LLM-as-a-Judge (GPT-4o, 1–5 scale): the tuned Whisper Large + Llama3 model averaged 2.328, compared with 2.145 for GPT-4o-Audio and 1.426 for the Wav2vec2 Base + Llama 3 cascade. In pairwise comparisons it won 96.55% of the time against the cascade and 80.78% against GPT-4o-Audio.
  • Human evaluation: seven raters scored 12 samples on three criteria: how relevant the suggestions were (SR), how easy they were for users to understand (UU), and an overall assessment (OE). The tuned model scored 3.80, 3.81 and 3.73 respectively, versus 2.88, 3.51 and 3.07 for GPT-4o-Audio, which corresponds to gains of 24.2%, 8.5% and 21.5%.
  • With the same audio encoder, Mistral-7B and Llama-3.1-8B backbones performed comparably after tuning. This points to task-specific instruction tuning as the main driver of the gains.
  • Limitations: the setup covers only read-aloud practice and gives text-only feedback. Free conversation and spoken feedback are left for future work. The code is released at github.com/zoeyada/ALMs4Learning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore180 works · 34,304 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSpeech & spoken language · 480 works

Cite

Cite as Hongfu Liu, Zhouying Cui, Xiangming Gu and Ye Wang. 2026. Unlocking Large Audio-Language Models for Interactive Language Learning. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.190

@inproceedings{liu2026unlocking,
  title = {{Unlocking Large Audio-Language Models for Interactive Language Learning}},
  author = {Hongfu Liu, Zhouying Cui, Xiangming Gu and Ye Wang},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.190},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.190}
}

References

39 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Moustafa Amrate and Pi-hua Tsai. 2024. Computer-assisted pronunciation training: A systematic review. ReCALL, pages 1–21. find
  2. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 421…
  3. Vipul Arora, Aditi Lahiri, and Henning Reetz. 2018. Phonological feature-based speech recognition system for pronunciation training in non-native language learning. The Journal of the Acoustical Society of America, 143(1):98–108. find
  4. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Preprint, arXiv:2006.11477.
  5. Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Pan Zhou, Yao Wan, and Lichao Sun. 2024. Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. Preprint, arXiv:2402.04788.
  6. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.
  7. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919.
  8. Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. 2023. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems, 36:18090–18108. find
  9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and Abhishek et al. Kadian. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Maxine Eskenazi. 2009. An overview of spoken language technology for education. Speech Communication, 51(10):832–844. find
  11. Yiqing Feng, Guanyu Fu, Qingcai Chen, and Kai Chen. 2020. Sed-mdd: Towards sentence dependent end-to-end mispronunciation detection and diagnosis. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3492–3496. find
  12. Xuelong Geng, Tianyi Xu, Kun Wei, Bingshen Mu, Hongfei Xue, He Wang, Yangze Li, Pengcheng Guo, Yuhang Dai, Longhao Li, Mingchen Shao, and Lei Xie. 2024. Unveiling the potential of llm-based asr on chinese open-source datasets. In 2024 IEEE 14th International Symposium on Chinese Spoken Language Proc… find
  13. Yuan Gong, Ziyi Chen, Iek-Heng Chu, Peng Chang, and James Glass. 2022. Transformer-based multi-aspect multi-granularity non-native english speaker pronunciation assessment. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7262–7266. IEEE.
  14. J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685.
  15. Wenping Hu, Yao Qian, Frank K Soong, and Yong Wang. 2015. Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers. Speech Communication, 67:154–166. find
  16. Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. 2024. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38… find
  17. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  18. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  19. Yassine El Kheir, Ahmed Ali, and Shammur Absar Chowdhury. 2023. Automatic pronunciation assessment–a review. arXiv preprint arXiv:2310.13974. find
  20. Daniel Korzekwa, Jaime Lorenzo-Trueba, Szymon Za-porowski, Shira Calamaro, Thomas Drugman, and Bozena Kostek. 2021. Mispronunciation detection in non-native (l2) english with uncertainty modeling. In ICASSP 2021-2021 IEEE international conference on acoustics, speech and signal processing (ICASSP),…
  21. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  22. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.
  23. Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. find
  24. Hongfu Liu, Hengguan Huang, and Ye Wang. 2024. Advancing test-time adaptation in wild acoustic test settings. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7138–7155. find
  25. Lynn E Henrichsen. 2021. An illustrated taxonomy of online capt resources. RELC Journal, 52(1):179– 188. find
  26. Hongfu Liu, Mingqian Shi, and Ye Wang. 2023b. Zero-shot automatic pronunciation assessment. arXiv preprint arXiv:2305.19563. — [arXiv:2305.19563](https://arxiv.org/abs/2305.19563) find
  27. Ziyang Ma, Yifan Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, and Xie Chen. 2024. An embarrassingly simple approach for llm with strong asr capacity. ArXiv, abs/2402.08846. — [arXiv:2402.08846](https://arxiv.org/abs/2402.08846) find
  28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  29. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
  30. Linkai Peng, Kaiqi Fu, Binghuai Lin, Dengfeng Ke, and Jinsong Zhang. 2021. A study on fine-tuning wav2vec2. 0 model for the task of mispronunciation detection and diagnosis. In Interspeech, pages 4448– 4452. find
  31. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. Preprint, arXiv:2212.04356.
  32. Pamela M Rogerson-Revell. 2021. Computer-assisted pronunciation training (capt): Current issues and future directions. RELC Journal, 52(1):189–205. find
  33. Silke M Witt and Steve J Young. 2000. Phone-level pronunciation scoring and assessment for interactive language learning. Speech communication, 30(2-3):95–108. find
  34. Xiaoshuo Xu, Yueteng Kang, Songjun Cao, Binghuai Lin, and Long Ma. 2021. Explore wav2vec 2.0 for mispronunciation detection. In Interspeech, pages 4428–4432. find
  35. Mu Yang, Kevin Hirschi, Stephen D Looney, Okim Kang, and John HL Hansen. 2022. Improving mispronunciation detection with wav2vec2-based momentum pseudo-labeling for accentedness and intelligibility assessment. arXiv preprint arXiv:2203.15937. — [arXiv:2203.15937](https://arxiv.org/abs/2203.15937) find
  36. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000.
  37. Junbo Zhang, Zhiwen Zhang, Yongqing Wang, Zhiyong Yan, Qiong Song, Yukai Huang, Ke Li, Daniel Povey, and Yujun Wang. 2021. speechocean762: An open-source non-native english speech corpus for pronunciation assessment. arXiv preprint arXiv:2104.01378.
  38. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
  39. Guanlong Zhao, Evgeny Chukharev-Hudilainen, Sinem Sonsaat, Alif Silpachai, Ivana Lucic, Ricardo Gutierrez-Osuna, and John Levis. 2018. L2-arctic: A non-native english speech corpus. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-unlocking-large-audio-language-models/