Elamur Research⌕
☰
home / papers / 2025 / Exploiting the English Vocabulary Profile for L2 word-level vocabulary assessment with LLMs

Exploiting the English Vocabulary Profile for L2 word-level vocabulary assessment with LLMs

Paper · 2025 · Stefano Bannò, Kate M. Knill and Mark J. F. Gales

Summary

The paper addresses fine-grained L2 vocabulary assessment. Instead of scoring whole essays or tagging words by part of speech alone, it gives each word a CEFR level based on the sense it carries in its sentence. The authors pair LLMs with the English Vocabulary Profile (EVP), a public resource built from the Cambridge Learner Corpus that links words, phrases, idioms and collocations to CEFR levels. Polysemy and multi-word expressions make this hard.

The study has three main experiments:

  1. LLMs pick the intended EVP sense of polysemous words, tested on the EVP's own learner example sentences.
  2. Learner sentences are annotated word by word with EVP levels, and LLMs are compared against a random baseline and a PoS-based baseline.
  3. The predicted word levels are applied to essay-level datasets (EFCAMDAT, ELLIPSE) to see how well they track overall proficiency.

The method is also turned around to check whether the EVP's own levels are internally consistent. LLMs, especially Qwen 2.5 32B, beat the PoS baseline because they draw on semantic context. The work offers a replicable route to vocabulary feedback inside automated essay grading.

Key points

  • The British English part of the EVP contains 15,671 entries covering 6747 unique words, so many words have several sense- or function-specific entries.
  • On EFCAMDAT holistic score prediction, Qwen-derived vocabulary features beat PoS-based features:
    • with a naive classifier, PCC is 0.636 vs 0.580;
    • with an SVR, PCC is 0.771 vs 0.734 (SRC 0.749 vs 0.713).
  • On ELLIPSE, a Qwen-SVR trained only on holistic scores reaches PCC 0.650 with the overall score. Among the analytic scores, its predictions match Vocabulary (0.637) and Phraseology (0.630) most closely.
  • An EVP consistency check on the polysemous words work and like in EFCAMDAT gives accuracies of 88.6–99.4%, depending on the threshold.
  • The authors suggest a hybrid system: an LLM handles ambiguous words and the PoS-based model handles unambiguous ones. They also plan to release their word-level EVP annotations.

Interesting ideas

  • I1. Grade a word by asking an LLM which entry of a curated sense inventory fits the sentence, then read the level off that entry; this keeps the output tied to an inspectable resource instead of a free-form guess.
  • I2. Send only genuinely ambiguous items to the expensive LLM and let a cheap PoS-based lookup handle words with a single reading.
  • I3. Score a level label as correct if it appears "at its level or higher," since the evaluation treats vocabulary use as a lower bound on proficiency.
  • I4. Run the trained predictor backwards over large unlabeled data to audit the reference resource itself, checking whether its labels behave sensibly in the wild.
  • I5. Check what a feature set really captures by seeing which analytic sub-score it correlates with most, even when it was trained only on a holistic target.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): The same pattern could map content words to WordNet/FrameNet senses, with an LLM selecting among listed candidates so every sense assignment points to a concrete ontology entry.
  • I2 (learn): LLM teacher tiers could be invoked only where the lexicon lists several senses or frames, saving cost while the deterministic path covers the rest.
  • I3 (answer): When Mova grades reading texts or student answers by difficulty, cumulative threshold metrics would avoid punishing advanced readers for using easy words.
  • I4 (learn): Silver predictions over big corpora could flag suspicious entries in compiled lexicons (verb classes, sense levels) for repair-rule review.
  • I5 (learn): Mova evaluations could test that a component (e.g., coreference features) aligns with its intended sub-skill more than with unrelated ones, acting as a built-in control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom256 works · 123,373 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksDevelopmental and Educational Psychology · 523 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,379 worksGrammatical error correction & learner language · 162 works

Cite

Cite as Stefano Bannò, Kate M. Knill and Mark J. F. Gales. 2025. Exploiting the English Vocabulary Profile for L2 word-level vocabulary assessment with LLMs. Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.bea-1.45

@inproceedings{banno2025exploiting,
  title = {{Exploiting the English Vocabulary Profile for L2 word-level vocabulary assessment with LLMs}},
  author = {Stefano Bannò, Kate M. Knill and Mark J. F. Gales},
  booktitle = {Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025)},
  year = {2025},
  doi = {10.18653/v1/2025.bea-1.45},
  url = {https://doi.org/10.18653/v1/2025.bea-1.45}
}

References

42 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Desislava Aleksandrova and Vincent Pouliot. 2023. CEFR-based contextual lexical complexity classifier in English and French. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 518–527, Toronto, Canada. Association for Computational Linguistics. ↗
  2. David Alfter, Yuri Bizzoni, Anders Agebjörn, Elena Volodina, and Ildikó Pilán. 2016. From distributions to labels: A lexical proficiency analysis using learner corpora. In Proceedings of the joint workshop on NLP for Computer Assisted Language Learning and NLP for Language Acquisition, pages 1–7, Umeå, Sweden. LiU Electronic Press.
  3. David Alfter and Elena Volodina. 2018. Towards single word lexical complexity prediction. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 79–88, New Orleans, Louisiana. Association for Computational Linguistics. find
  4. Richard C. Anderson and Peter Freebody. 1981. Vocabulary knowledge. In J. P. Guthrie, editor, Comprehension and teaching: Research reviews, pages 77–117. International Reading Association, Newark. find
  5. Richard C. Anderson and Peter Freebody. 1983. Reading comprehension and the assessment and acquisition of word knowledge. In B. Hutson, editor, Advances in reading / language research, volume 2. JAI Press, Greenwich. find
  6. Yuki Arase, Satoru Uchida, and Tomoyuki Kajiwara. 2022. CEFR-based sentence difficulty annotation and assessment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6206–6219, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  7. Melissa M. Baese-Berk, Shiloh Drake, Kurtis Foster, Dae-yong Lee, Cecelia Staggs, and Jonathan M. Wright. 2021. Lexical diversity, lexical sophistication, and predictability for speech in multiple listening conditions. Frontiers in psychology, 12:661415. ↗
  8. Stefano Bannò, Hari Krishna Vydana, Kate Knill, and Mark Gales. 2024. Can GPT-4 do L2 analytic assessment? In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 149–164, Mexico City, Mexico. Association for Computational Linguistics. ↗
  9. Stephen Bax. 2012. Text inspector. Online text analysis tool. In our first experiment specifically, learner example sentences from the EVP may contain errors, but not involving the target word. Additionally, this experiment does not require lemmatisation. find
  10. Annette Capel. 2015. The English Vocabulary Profile. In J. Harrison and F. Barker, editors, English Profile in Practice. Cambridge University Press, Cambridge. find
  11. Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge University Press, Cambridge. find
  12. Shi Huawei and Vahid Aryadoust. 2023. A systematic review of automated writing evaluation systems. Education and Information Technologies, 28(1):771– 795. ↗
  13. Council of Europe. 2020. Common European Framework of Reference for Languages: Learning, Teaching, Assessment - Companion volume. Council of Europe, Strasbourg. find
  14. Kristopher Kyle, Scott Crossley, and Cynthia Berger. 2018. The tool for the automatic analysis of lexical sophistication (TAALES): version 2.0. Behavior research methods, 50(3):1030–1046. ↗
  15. Scott Crossley, Yu Tian, Perpetual Baffour, Alex Franklin, Youngmeen Kim, Wesley Morris, Meg Benner, Aigner Picou, and Ulrich Boser. 2023. The English Language Learner Insight, Proficiency and Skills Evaluation (ELLIPSE) Corpus. International Journal of Learner Corpus Research, 9(2):248–269. ↗
  16. Kristopher Kyle and Scott A Crossley. 2015. Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly, 49(4):757–786. ↗
  17. Scott A Crossley and Danielle S McNamara. 2012. Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35(2):115–135. find
  18. Luise Dürlich and Thomas François. 2018. EFLLex: A graded lexical resource for learners of English as a foreign language. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  19. Núria Gala, Thomas François, Delphine Bernhard, and Cédrick Fairon. 2014. Un modèle pour prédire la complexité lexicale et graduer les mots. In TALN’2014, pages 91–102. find
  20. Aina Garí Soler and Marianna Apidianaki. 2021. Let’s play monopoly: BERT can reveal words’ polysemy level and partitionability into senses. Transactions of the Association for Computational Linguistics, 9:825– 844. find
  21. Atta Gebril and Lia Plakans. 2016. Source-based tasks in academic writing assessment: Lexical diversity, textual borrowing and proficiency. Journal of English for Academic Purposes, 24:78–88. ↗
  22. Jeroen Geertzen, Theodora Alexopoulou, and Anna Korhonen. 2013. Automatic linguistic annotation of large scale L2 databases: The EF-Cambridge Open Language Database (EFCAMDAT). In Proceedings of the 31st Second Language Research Forum, pages 240–254, Somerville. Cascadilla Proceedings Project. find
  23. Batia Laufer and Paul Nation. 1995. Vocabulary size and use: Lexical richness in L2 written production. Applied Linguistics, 16(3):307–322. ↗
  24. Paul Lennon. 2008. Contrastive analysis, error analysis, interlanguage. In S. Gramley and V. Gramley, editors, Bielefeld Introduction to Applied Linguistics. A Course Book. Bielefeld, pages 51–60. Aisthesis, Bielefeld. find
  25. Agnieszka Leńko-Szymańska. 2015. The English Vocabulary Profile as a benchmark for assigning levels to learner corpus data. In M. Callies and S. Götz, editors, Learner Corpora in Language Testing and Assessment. John Benjamins, Amsterdam; Philadelphia. ↗
  26. Adian Liusie, Potsawee Manakul, and Mark Gales. 2024. LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 139–151, St. Julian’s, Malta. Association for Computational Linguistics. ↗
  27. Llama Team. 2024. The Llama 3 herd of models.
  28. Xiaofei Lu. 2012. The relationship of lexical richness to the quality of ESL learners’ oral narratives. The Modern Language Journal, 96(2):190–208. ↗
  29. Paul Meara and Barbara Buxton. 1987. An alternative to multiple choice vocabulary tests. Language Testing, 4(2):142–154. ↗
  30. Paul Nation. 1983. Testing and teaching vocabulary. Guidelines, 5(1):12–25. find
  31. Liz Hamp-Lyons. 1995. Rating nonnative writing: The trouble with holistic scoring. TESOL Quarterly, 29(4):759–762. ↗
  32. Diane Nicholls. 2003. The Cambridge Learner Corpus: Error coding and analysis for lexicography and ELT. In Proceedings of the Corpus Linguistics 2003 Conference, pages 572–581. find
  33. Luke Harding. 2014. Communicative language testing: Current issues and future research. Language Assessment Quarterly, 11(2):186–197. ↗
  34. OpenAI. 2023. GPT-4 Technical Report. Preprint, arXiv:2303.08774. ↗
  35. Yan Huang, Jeroen Geertzen, Rachel Baker, Anna Korhonen, and Theodora Alexopoulou. 2017. The EF Cambridge Open Language Database (EFCAMDAT): Information for users. find
  36. John Read. 2000. Assessing vocabulary. Cambridge University Press, Cambridge. find
  37. John Read. 2013. Second language vocabulary assessment. Language Teaching, 46(1):41–52. ↗
  38. Norbert Schmitt, Diane Schmitt, and Caroline Clapham. 2001. Developing and exploring the behaviour of two new versions of the Vocabulary Levels Test. Language Testing, 18(1):55–88. No. of options 1 2 3 4 5 6 > 6 ↗
  39. Burr Settles, Geoffrey T. LaFlair, and Masato Hagiwara. 2020. Machine learning–driven language assessment. Transactions of the Association for computational Linguistics, 8:247–263. ↗
  40. Yukio Tono. 2013. The CEFR-J handbook: A resource book for using CAN-DO descriptors for English language teaching. Tokyo: Taishukan. find
  41. Jeanine Treffers-Daller, Patrick Parslow, and Shirley Williams. 2018. Back to basics: How measures of lexical diversity can help discriminate between cefr levels. Applied Linguistics, 39(3):302–327. find
  42. Satoru Uchida and Masashi Negishi. 2018. Assigning CEFR-J levels to English texts based on textual features. In Proceedings of Asia Pacific Corpus Linguistics Conference, volume 4, pages 463–467. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/banno-2025-exploiting-english-vocabulary-profile-l2/