Elamur Research⌕
☰
home / papers / 2026 / Cross-Corpus CEFR Classification through Artificial Learners Perplexities

Cross-Corpus CEFR Classification through Artificial Learners Perplexities

Summary

The paper asks whether classic statistical classifiers with well-designed features can match fine-tuned transformers and prompted LLMs at predicting the CEFR level of English learner essays. It focuses on the case where the test corpus differs from the training corpus. The authors use language models as feature generators rather than as classifiers. One model is trained on native language, and six "artificial learner" models are trained on EFCAMDAT texts tied to specific proficiency levels. Each model scores an essay, and the resulting perplexities show how surprising the learner's wording is compared with native usage or with a given level. Statistical classifiers use these features. They are tested on an EFCAMDAT hold-out and on two external corpora, KUPA-KEYS and CELVA-SP. The external tests are run zero-shot and also after retraining only the classifier on 90% of each target corpus. On KUPA-KEYS the approach beats fine-tuned BERT and prompting even without adaptation. On CELVA-SP, zero-shot transfer is weak, but in-domain classifier training recovers much of the gap. The results suggest that interpretable, transferable features are a robust alternative to end-to-end models for proficiency assessment.

Key points

  • Data: The refined EFCAMDAT (Shatz, 2020) contains 723,282 texts from Englishtown schools. It is divided into three parts: artificial-learner training, classifier fitting, and in-domain hold-out. CELVA-SP and KUPA-KEYS are used only for evaluation. CELVA-SP consists of ESP writing by French undergraduates, produced in a 45-minute in-class task with three prompts.
  • KUPA-KEYS: RMSE is 0.707 zero-shot and 0.660 with the 90-10 split. Both results outperform fine-tuned BERT and prompt-based LLMs.
  • CELVA-SP: Zero-shot RMSE is 1.437, worse than the LLM's 1.016. The 90-10 setting reduces it to 0.872. The authors present this as the first comprehensive benchmark for this corpus.
  • Average across the three evaluation sets (90-10 split): Macro F1 is 0.446 for the perplexity-based models, compared with 0.287 for BERT and 0.175 for prompting.
  • Constraints and release: Model size was restricted because training the artificial learners was done on just one GPU with 16GB of memory. The models pre-trained on EFCAMDAT are planned for release on the server that hosts EFCAMDAT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2bvpcczgao2t

@inproceedings{stearns2026cross,
  title = {{Cross-Corpus CEFR Classification through Artificial Learners Perplexities}},
  author = {Stearns, Bernardo and McCrae, John P. and Gaillat, Thomas},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2bvpcczgao2t},
  url = {https://doi.org/10.63317/2bvpcczgao2t}
}

References

20 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. KV Aditya Srivatsa, Kaushal Kumar Maurya, and Ekaterina Kochmar. 2025. Can LLMs Reliably Simulate Real Students’ Abilities in Mathematics and Reading Comprehension? arXiv e-prints, page arXiv:2507.08232. find
  2. Tatsuya Aoyama and Nathan Schneider. 2024. Modeling nonnative sentence processing with L2 language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4927–4940, Miami, Florida, USA. Association for Computational Linguistics. find
  3. Luca Benedetto, Giovanni Aradelli, Antonia Donvito, Alberto Lucchetti, Andrea Cappelli, and Paula Buttery. 2024. Using LLMs to simulate students’ responses to exam questions. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11351–11368. find
  4. Luca Benedetto, Gabrielle Gaudeau, Andrew Caines, and Paula Buttery. 2025. Assessing how accurately large language models encode and apply the common european framework of reference for languages. Computers and Education: Artificial Intelligence, 8:100353. find
  5. Thomas Gaillat, Andrew Simpkin, Nicolas Ballier, Bernardo Stearns, Annanda Sousa, Manon Bouyé, and Manel Zarrouk. 2022. Predicting CEFR levels in learners of English: The use of microsystem criterial features in a machine learning approach. ReCALL, 34(2):130–146. find
  6. Jeroen Geertzen, Theodora Alexopoulou, Anna Korhonen, et al. 2013. Automatic linguistic annotation of large scale l2 databases: The ef-cambridge open language database (efcamdat). In Proceedings of the 31st Second Language Research Forum. Somerville, MA: Cascadilla Proceedings Project, pages 240–254… find
  7. Elma Kerz, Daniel Wiechmann, Yu Qiao, Emma Tseng, and Marcus Ströbel. 2021. Automated classification of written proficiency levels on the CEFR-scale through complexity contours and RNNs. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 199–209… find
  8. Wonbin Kim. 2024. Let’s make an artificial learner to analyze learners’ language! , (70):167–193. find
  9. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM computing surveys, 55(9):1–35. find our card
  10. Cyriel Mallart, Andrew Simpkin, Rémi Venant, Nicolas Ballier, Bernardo Stearns, Jen Yu Li, and Thomas Gaillat. 2023. A new learner language data set for the study of English for Specific Purposes at university level. In Proceedings of the 4th Conference on Language, Data and Knowledge - LDK 2023, vo… find
  11. Elijah Mayfield and Alan W Black. 2020. Should you fine-tune BERT for automated essay scoring? In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 151–162, Seattle, WA, USA → Online. Association for Computational Linguistics. find
  12. Jay Mohta, Kenan Ak, Yan Xu, and Mingwei Shen. 2023. Are large language models good annotators? In Proceedings on "I Can’t Believe It’s Not Better: Failure Modes in the Age of Foundation Models" at NeurIPS 2023 Workshops, volume 239 of Proceedings of Machine Learning Research, pages 38–48. PMLR. find
  13. Itamar Shatz. 2020. Refining and modifying the EF-CAMDAT: Lessons from creating a new corpus from an existing large-scale English learner language database. International Journal of Learner Corpus Research, 6(2):220–236. find
  14. Bernardo Stearns, Nicolas Ballier, Thomas Gaillat, Andrew Simpkin, and John P McCrae. 2024. Evaluating the generalisation of an artificial learner. In Swedish Language Technology Conference and NLP4CALL, pages 199–208. find
  15. David Strohmaier and Paula Buttery. 2024. Semantic error prediction: Estimating word production complexity. In Proceedings of the 13th Workshop on Natural Language Processing for Computer Assisted Language Learning, pages 209–225, Rennes, France. LiU Electronic Press. find
  16. Hakyung Sung and Kristopher Kyle. 2024. Leveraging pre-trained language models for linguistic analysis: A case of argument structure constructions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7302–7314, Miami, Florida, USA. Association for Computa… find
  17. Satoru Uchida and Masashi Negishi. 2025. Assigning CEFR-J levels to English learners’ writing: An approach using lexical metrics and generative AI. Research Methods in Applied Linguistics, 4(1):100199. find
  18. Georgios Velentzas, Andrew Caines, Rita Borgo, Erin Pacquetet, Clive Hamilton, Taylor Arnold, Diane Nicholls, Paula Buttery, Thomas Gaillat, Nicolas Ballier, et al. 2024. Logging keystrokes in writing by English learners. In Proceedings of the 2024 Joint International Conference on Computational Lin… find
  19. Songlin Xu, Xinyu Zhang, and Lianhui Qin. 2024. EduAgent: Generative student agents in learning. find
  20. Victoria Yaneva, King Yiu Suen, Le An Ha, Janet Mee, Milton Quranda, and Polina Harik. 2024. Automated scoring of clinical patient notes: Findings from the Kaggle competition and their translation into practice. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Ap… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stearns-2026-cross-corpus-cefr-classification-through/