Elamur Research⌕
☰
home / papers / 2026 / MASA: A Novel Multimodal Foundation Model for L2 Speaking Assessment in Picture-description Scenarios

MASA: A Novel Multimodal Foundation Model for L2 Speaking Assessment in Picture-description Scenarios

Summary

MASA is a scoring model for automatic speaking assessment (ASA) of second-language (L2) learners in picture-description tasks. In these tasks, a test-taker sees an image and several open questions and then answers aloud. Earlier ASA systems depend on handcrafted features for fluency, pronunciation, grammar and content, which take a lot of effort to build. They also ignore the rubrics that human raters actually use.

MASA replaces that feature engineering with the pre-trained audio and vision encoders of the Phi4-Multimodal-Instruct foundation model. Speech, image and text tokens are mapped into one shared space. The model also receives a prompt containing the scoring rubrics, the questions, an instruction and an ASR transcript of the answer. It is trained with two objectives: generating the score as text and predicting it with an auxiliary classifier. The authors add a training-free debiasing step that compares output distributions from multimodal and single-modality inputs. This reduces the model's reliance on unimodal priors.

On the GEPT picture-description data, MASA is reported to beat several strong baselines. This suggests that multimodal foundation models are a practical route to rubric-aware spoken-language scoring.

Key points

  • The task is holistic proficiency classification on a 0–5 scale, using GEPT (General English Proficiency Test) picture-description responses with questions such as Q1–Q4 about an image.
  • The backbone is Phi4-Multimodal-Instruct, with its audio and vision encoders and projectors. It is adapted through parameter-efficient LoRA tuning, using separate Speech LoRA and Vision LoRA modules.
  • Learner speech is transcribed with Whisper-large-v2. The prompt combines the rubric guidelines, the questions, an instruction and the transcript, and log-Mel filter-bank features are fed to the audio encoder.
  • The training loss combines a classification cross-entropy term with an autoregressive generation term, balanced by a weight α ∈ [0,1]. At inference, the lightweight classifier is the main scorer.
  • The training-free debiasing method contrasts predictions made from full multimodal input with those made from single-modal input, which reduces scoring bias. The provided excerpt does not include the exact metric values.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2yc5vwhcevz5

@inproceedings{yan2026masa,
  title = {{MASA: A Novel Multimodal Foundation Model for L2 Speaking Assessment in Picture-description Scenarios}},
  author = {Yan, Bi-Cheng and Chao, Fu-An and Lin, Hong-Yun H.Y. and Chen, Berlin},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2yc5vwhcevz5},
  url = {https://doi.org/10.63317/2yc5vwhcevz5}
}

References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelrahman Abouelenin et al. 2025. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. in arXiv preprint arXiv:2503.01743. find our card
  2. Vedika Agarwal, Rakshith Shetty, and Mario Fritz. 2020. Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages. 9690– 9698. find
  3. Stefano Bannò and Marco Matassoni. 2022. Proficiency assessment of L2 spoken English using wav2vec 2.0. In Proceedings of the IEEE Spoken Language Technology Workshop (SLT), pages 1088–1095. find
  4. Lei Chen, Keelan Evanini, and Xie Sun. 2010. Assessment of non-native speech using vowel space characteristics. In Proceedings of the IEEE Spoken Language Technology Workshop (SLT), pages 139–144. find
  5. Nancy F. Chen and Haizhou Li. 2016. Computer-assisted pronunciation training: From pronunciation scoring towards spoken language learning. In Proceedings of Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA). find
  6. Tri Dao. 2024. FlashAttention-2: Faster attention with better parallelism and work partitioning. In Proceedings of International Conference on Learning Representations (ICLR). find
  7. Keelan Evanini and Xinhao Wang. 2013. Automated speech scoring for Nonnative middle school students with multiple task types. In Proceedings of INTERSPEECH (Interspeech), pages 2435–2439, 2013. find
  8. Keelan Evanini, Maurice Cogan Hauck, Kenji Hakuta. 2017. Approaches to automated scoring of speaking for K–12 English language proficiency assessments. In ETS Research Report Series, pages 1–11, 2017. find
  9. Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. 2025. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. In arXiv preprint arXiv:2507.0… — [arXiv:2507.08128](https://arxiv.org/abs/2507.08128) find
  10. Team Gemma et al. 2025. Gemma 3 technical report. In arXiv preprint arXiv:2503.19786. find
  11. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In Proceedings of International Conference on Learning Representations (ICLR). find
  12. Lei Huang et al. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. In ACM Transactions on Information Systems, vol. 43, pages. 1–55. find our card
  13. Weiquan Huang et al. 2024. Llm2clip: Powerful language model unlocks richer visual representation. In arXiv preprint arXiv:2411.04997. — [arXiv:2411.04997](https://arxiv.org/abs/2411.04997) find
  14. Shin'ichiro Ishikawa. 2011. A new horizon in learnercorpus studies: The aim of ICNALE project.Corpora and language technologies in teaching,learning and research, G. Weir, S. Ishikawa, and K.Poonpon, Eds., pages. 3–11. find
  15. Davis Larry and John M. Norris. 2024. Challenges and Innovations in Speaking Assessment. Taylor & Francis. find
  16. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13872… find our card
  17. Jiun-Ting Li, Tien-Hong Lo, Bi-Cheng Yan, Yung-Chang Hsu and Berlin Chen. 2023. Graph-enhanced Transformer architecture with novel use of CEFR vocabulary profile and filled pauses in automated speaking assessment. In Proceedings of the workshop on Speech and Language Technology in Education (SLaTE),… find
  18. Hong-Yun Lin, Tien Hong Lo, Yu Hsuan Fang, Jhen-Ke Lin, Chung-Chun Wang, Hao-Chien Lu, Berlin Chen. 2025. The NTNU system at the S&I Challenge 2025 SLA Open Track. In the Workshop on Speech and Language Technology in Education (SLaTE). find
  19. Russell Moore, Andrew Caines, Calbert Graham, and Paula Buttery. 2015. Incremental dependency parsing and disfluency detection in spoken learner English. In Proceedings of the International Conference on Text, Speech and Dialogue (TSD), pages 470–479. find
  20. Wen-Hsuan Peng, Sally Chen, and Berlin Chen. 2024. Enhancing automatic speech assessment leveraging heterogeneous features and soft labels for ordinal classification. In IEEE Spoken Language Technology Workshop (SLT), pages 945–952. find
  21. Team Qwen et al. 2024. Qwen2.5 technical report. in arXiv preprint. find
  22. Mengjie Qian, Kate Knill, Stefano Banno, Siyuan Tang, Penny Karanasou, Mark J.F. Gales, and Diane Nicholls. 2024. Speak & improve challenge 2025: Tasks and baseline systems. In arXiv preprint arXiv:2412.11985. find
  23. Yao Qian, Patrick Lange, Keelan Evanini, Robert Pugh, Rutuja Ubale, Matthew Mulholland, and Xinhao Wang. 2019. Neural approaches to automated speech scoring of monologue and dialogue responses. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages… find
  24. Yao Qian, Rutuja Ubale, Matthew Mulholland, Keelan Evanini, and Xinhao Wang. 2018. A prompt-aware neural network approach to content-based scoring of nonnative spontaneous speech. In Proceedings of IEEE Spoken Language Technology Workshop (SLT), pages 979–986. find
  25. Yaman Kumar Singla, Avykat Gupta, Shaurya Bagga, Changyou Chen, Balaji Krishnamurthy, Rajiv Ratn Shah. 2021. Speaker-conditioned hierarchical modelling for automated speech scoring. In Proceedings of the ACM international conference on information & knowledge management (CIKM), pages 1681– 1691. find
  26. Yaman Kumar Singla, Jui Shah, Changyou Chen, and Rajiv Ratn Shah. 2022. What do audio transformers hear? probing their representations for language delivery structure. In Proceedings of IEEE International Conference on Data Mining Workshops (ICDMW), pages. 910–925. find
  27. Yike Wu, Yu Zhao, Shiwan Zhao, Ying Zhang, Xiaojie Yuan, Guoqing Zhao, and Ning Jiang. 2022. Overcoming language priors in visual question answering via distinguishing superficially similar instances. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 5721–57… find
  28. Klaus Zechner and Keelan Evanini. 2019. Automated speaking assessment: Using language technologies to score spontaneous speech. Routledge. find
  29. Klaus Zechner, Derrick Higgins, Xiaoming Xi, and David M. Williamson. 2009. Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, volume 51, page 883–895, 2009. find
  30. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975– 11986. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yan-2026-masa-novel-multimodal-foundation-model/