Elamur Research⌕
☰
home / papers / 2025 / Automatic Proficiency Assessment in L2 English Learners

Automatic Proficiency Assessment in L2 English Learners

Paper · 2025 · Armita Mohammadi, Alessandro Lameiras Koerich, Laureano Moro-Velazquez et al.

Summary

Human raters scoring English as a second language are expensive and inconsistent with each other and with themselves. This study tests whether deep learning can classify proficiency automatically from audio, from text, and from spontaneous interview dialogue. The authors move from scripted to unscripted material using three corpora: ANGLISH (read passages and monologues by native English and French speakers), EFCamDat (learner essays with CEFR levels) and a private set of recorded proficiency interviews. On the audio side they compare a 2D CNN, a CNN that convolves only along frequency, a ResNet and a fine-tuned wav2vec 2.0. On the text side they put a classifier on top of a frozen BERT. For dialogues they build a pipeline that removes silence, performs diarization and transcribes with Whisper. They also try joint prediction of proficiency and gender, and they compare ways of segmenting text input. Every split is speaker-independent. The authors report that the pretrained wav2vec 2.0 is the most promising model. The excerpt available here leaves out the exact scores.

Key points

  • ANGLISH: 67 speakers in three groups. NES are native speakers. FR1 are French speakers without phonetics training. FR2 are French students with formal phonetics training. The corpus holds over 5.5 h recorded in an anechoic chamber, and the read part is about 1.5 h covering 1,260 sentences.
  • EFCamDat: 1,180,310 essays from learners of more than 172 nationalities. Its 16 internal levels map to the CEFR scale A1–C1, and the metadata includes native language, age and gender.
  • Private interview data: an interviewer leads each dialogue. Raters score fluency, comprehension, accuracy and vocabulary, then give an overall score.
  • Audio front end:
    • resampled to 16 kHz;
    • cut into 8-second chunks for fixed-input models;
    • 40-bin Mel FBanks with z-score normalization.
  • Dialogue cleaning:
    • Pydub removes silence (500 ms minimum, −40 dB threshold, 100 ms padding).
    • PyAnnote produces the first diarization, which SpeechBrain then refines.
    • Whisper transcribes, with several variants compared.
  • wav2vec 2.0 fine-tuning: the convolutional feature extractor stays frozen while the transformer is tuned. Statistics pooling feeds two linear heads, and the encoder and the heads each get their own optimizer.

Interesting ideas

  • I1. All splits keep each speaker on only one side, so a model cannot get credit for recognizing a voice it already saw in training.
  • I2. In each interview, the participant with the most total speaking time is taken to be the learner, a cheap heuristic that tells the roles apart without any labels.
  • I3. A second, easy label (gender) is predicted alongside the main proficiency target, and this helped compared with single-task training.
  • I4. How the input text is split into segments is treated as a variable worth measuring, because the choice changes BERT classification results.
  • I5. The pretrained low-level extractor is frozen, and the upper layers and the new heads each get their own optimizer.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation sets should be split by source document or author rather than by sentence, so that repeated phrasing cannot inflate parser or QA scores.
  • I2 (frames): In dialogues from tales, the speaker with the most turns or the most text could serve as a default protagonist or addressee guess, recorded as an explicit and overridable rule.
  • I3 (tree): gparse could carry auxiliary heads for UPOS or morphological features next to the dependency head, which may regularize LAS training at little cost.
  • I4 (answer): When reading long fables or school texts, Mova should test windowing by sentence, paragraph or whole text rather than fixing one choice in advance.
  • I5 (learn): Mova could keep the lower transformer layers fixed and give the parser heads a separate learning rate, reducing overfitting on small silver treebanks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Armita Mohammadi, Alessandro Lameiras Koerich, Laureano Moro-Velazquez and Patrick Cardinal. 2025. Automatic Proficiency Assessment in L2 English Learners. arXiv:2505.02615.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.02615 · arXiv 2505.02615

@misc{mohammadi2025automatic,
  title = {{Automatic Proficiency Assessment in L2 English Learners}},
  author = {Armita Mohammadi, Alessandro Lameiras Koerich, Laureano Moro-Velazquez and Patrick Cardinal},
  year = {2025},
  doi = {10.48550/arxiv.2505.02615},
  eprint = {2505.02615},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.02615}
}

References

40 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. A. Chapelle and D. Douglas, Assessing Language through Computer Technology. Cambridge Language Assessment, Cambridge Univ Press, 2006. find
  2. J. C. Alderson, Assessing Reading. Cambridge Language Assessment, Cambridge Univ Press, 2000. find
  3. K. Zechner and K. Evanini, eds., Automated Speaking Assessment: Using Language Technologies to Score Spontaneous Speech. Routledge, 1st ed., 2019. find
  4. L. Chen, K. Zechner, S.-Y. Yoon, K. Evanini, X. Wang, A. Loukina, J. Tao, L. Davis, C. M. Lee, M. Ma, R. Mundkowsky, C. Lu, C. W. Leong, and B. Gyawali, “Automated scoring of nonnative speech using the speechratersm v. 5.0 engine,” ETS Research Report Series, vol. 2018, 03 2018. find
  5. A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” 2020.
  6. S. W. McKnight, A. Civelekoglu, M. J. Gales, S. Banno, A. Liusie, and K. M. Knill, “Automatic assessment of conversational speaking tests,” 2023. find
  7. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019.
  8. K. O’Shea and R. Nash, “An introduction to convolutional neural networks,” 2015. ↗
  9. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015. ↗
  10. R. Lado, Language Testing: The Construction and Use of Foreign Language Tests. Teacher’s Book. McGraw-Hill Book Comp, 1961. find
  11. S. P. Corder, “The significance of learner’s errors,” Intl Review of Applied Linguistics in Language Teaching, vol. 5, no. 1-4, pp. 161– 170, 1967. find
  12. S. Whyte, “Revisiting communicative competence in the teaching and assessment of language for specific purposes,” Language Education & Assessment, vol. 2, pp. 1–19, 04 2019. ↗
  13. M. Canale and M. Swain, “Theoretical bases of communicative approaches to second language teaching and testing,” Applied Linguistics, vol. I, 03 1980. ↗
  14. K. Nebhi and G. Szaszák, “Automatic assessment of spoken English proficiency based on multimodal and multitask transformers,” in 14th Intl Conf Recent Adv Natural Lang Process, pp. 769–776, 2023. find
  15. J. Pfingsthorn, Variability in Learner Errors as a Reflection of the CLT Paradigm Shift. Peter Lang, 2013, 2013. find
  16. J. A. Hawkins and P. Buttery, “Criterial features in learner corpora: Theory and illustrations,” English Profile J, vol. 1, p. e5, 2010. ↗
  17. C. Cortes and V. Vapnik, “Support-vector networks,” Machine learning, vol. 20, no. 3, pp. 273–297, 1995. ↗
  18. K. Zechner and I. Bejar, “Towards automatic scoring of non-native spontaneous speech,” in Human Lang Techn Conf NAACL, pp. 216– 223, 2006. find
  19. X. Xi, D. Higgins, K. Zechner, and D. Williamson, “Automated scoring of spontaneous speech using speechrater,” ETS Research Report Series, vol. 2008, 12 2008. find
  20. Y. Wang, M. Gales, K. Knill, K. Kyriakopoulos, A. Malinin, R. van Dalen, and M. Rashid, “Towards automatic assessment of spontaneous spoken english,” Speech Communication, vol. 104, pp. 47–56, 2018. ↗
  21. Y. Qian, K. Evanini, X. Wang, C. M. Lee, and M. D. Mulholland, “Bidirectional lstm-rnn for improving automated assessment of non-native children’s speech,” in Interspeech, 2017. find
  22. Y. K. Singla, A. Gupta, S. Bagga, C. Chen, B. Krishnamurthy, and R. R. Shah, “Speaker-conditioned hierarchical modeling for automated speech scoring,” in 30th ACM Intl Conf Inf & Knowledge Management, p. 1681–1691, 2021. find
  23. S. Bannò, K. Knill, M. Matassoni, V. Raina, and M. J. F. Gales, “L2 proficiency assessment using self-supervised speech representations,” ArXiv, vol. abs/2211.08849, 2022. ↗
  24. S. Bannò and M. Matassoni, “Proficiency assessment of l2 spoken english using wav2vec 2.0,” in IEEE Spoken Langu Techn Workshop, pp. 1088–1095, 2023.
  25. E. Islam, C. Park, and T. Hain, “Exploring speech representations for proficiency assessment in language learning,” in 9th Workshop on Speech and Language Technology in Education (SLaTE), pp. 151–155, 08 2023. find
  26. T.-H. Lo, F.-A. Chao, T.-I. Wu, Y.-T. Sung, and B. Chen, “An effective automated speaking assessment approach to mitigating data scarcity and imbalanced distribution,” 2025. find
  27. V. J. Schmalz and A. Brutti, “Automatic assessment of english cefr levels using bert embeddings,” in Italian Conference on Computational Linguistics, 2021.
  28. S. Bannò and M. Matassoni, “Cross-corpora experiments of automatic proficiency assessment and error detection for spoken english,” in 17th Works Innov Use NLP Building Educat Appl, (Seattle, Washington), pp. 82–91, 2022.
  29. J.-T. Li, B.-C. Yan, T.-H. Lo, Y.-C. Wang, Y.-C. Hsu, and B. Chen, “Automated speaking assessment of conversation tests with novel graph-based modeling on spoken response coherence,” 2024. find
  30. P. Muangkammuen and F. Fukumoto, “Multi-task learning for automated essay scoring with sentiment analysis,” in 1st Conf Asia-Pacific Chapter Assoc Comput Linguistics and 10th Intl Joint Conf Nat Lang Process, (Suzhou, China), pp. 116–123, 2020. find
  31. Y. Yang, J. Zhong, C. Wang, and Q. Li, “Exploring relevance and coherence for automated text scoring using multi-task learning,” in 34th Intl Conf Software Eng Knowledge Eng, pp. 323–328, 2022. find
  32. J. Robert, M. Webbie, et al., “Pydub,” 2018. find
  33. H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M.-P. Gill, “pyannote.audio: neural building blocks for speaker diarization,” 2019. find
  34. M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, “SpeechBrain: A general-purpose speech toolkit,” 2021. arXiv:2106.04624. ↗
  35. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022. ↗
  36. A. Tortel, “Anglish: Une base de données comparatives de l’anglais répété et parlé en l1 & l2,” TIPA. Travaux interdisciplinaires sur la parole et le langage, vol. 27, pp. 111–122, 06 2008. find
  37. P. Boersma and D. Weenink, “Praat: doing phonetics by computer (version 5.1.13),” 2009. find
  38. B. Bel and P. Blache, “The Resource Center for Oral Description (CRDO),” Interdisciplinary work of the Speech and Language Laboratory, vol. 25, pp. 13–18, 2006. find
  39. J. Geertzen, T. Alexopoulou, and A. Korhonen, “Automatic linguistic annotation of large scale l2 databases: The ef-cambridge open language database(efcamdat),” 2014. find
  40. Z. Zhou, S. Vajjala, and S. V. Mirnezami, “Experiments on non-native speech assessment and its consistency,” in 8th Workshop NLP Computer Assisted Language Learning, pp. 86–92, 2019. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mohammadi-2025-automatic-proficiency-assessment-l2-english/