Elamur Research⌕
☰
home / papers / 2026 / Language Proficiency as a Recoverable Dimension in Multilingual LLM Embeddings

Language Proficiency as a Recoverable Dimension in Multilingual LLM Embeddings

Paper · 2026 · Rodrigo Wilkens

Summary

The study asks whether multilingual LLMs hold CEFR proficiency as genuine internal structure, or whether their success at predicting proficiency rests on shallow cues such as text length or word frequency. The author uses the UniversalCEFR benchmark, which covers 13 languages and all six levels from A1 to C2. Text-only inputs go through frozen EuroLLM models in base and instruction-tuned variants at 1.7B and 9B parameters, and embeddings are taken from the final hidden layer.

There are two analyses. The first trains linear and non-linear probes on the embeddings to predict CEFR level. The second examines how the levels are arranged geometrically.

Probe results show that untuned embeddings reliably support CEFR classification. Differences in performance track the amount of annotated data and the language family more closely than model size or instruction tuning. Geometrically, the levels fall along one ordered direction. Distances between levels grow with their ordinal gap, and neighbouring levels overlap. This suggests proficiency is a continuous gradient rather than a set of arbitrary clusters. Such a gradient could support level-controlled text adaptation, calibrated feedback and alignment with teaching scales.

Key points

  • Data: UniversalCEFR, covering 13 languages and levels A1–C2. Corpora include cefr, elg and icle500 (English), caes and kwiqiz (Spanish), readme (French, Hindi), and cople2 and peapl2 (Portuguese).
  • Models: Only the EuroLLM family is used, so that architecture and training data stay controlled. Base and instruction-tuned variants are compared at 1.7B and 9B. All models are frozen, embeddings are extracted with HuggingFace, and no prompts or instructions are added.
  • Probes and evaluation: Logistic Regression and a linear SVM test linear separability, while an MLP tests for non-linear signal. Training uses stratified N-fold cross-validation, and results are scored with accuracy, macro F1, QWK and MAE.
  • Score range: Results vary widely by corpus. For the 1.7B instruction-tuned setting:
    • caes reaches about 0.99 accuracy.
    • English cefr scores 0.53–0.58 accuracy.
    • The French and Hindi readme subsets score roughly 0.34–0.45.
  • Geometry: Levels line up along a consistent ordered axis, with separation scaling with level distance (A1 vs. C2 lies farther apart than B1 vs. B2). This supports reading CEFR as a continuous latent dimension.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ExeterExeter · United Kingdom13 works · 203 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksMultilingual & cross-lingual NLP · 882 works

Cite

Cite as Rodrigo Wilkens. 2026. Language Proficiency as a Recoverable Dimension in Multilingual LLM Embeddings. Proceedings of the Joint Workshop on Readability and Text Simplification (READIxTSAR) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3zrv8qescsdg

@inproceedings{wilkens2026language,
  title = {{Language Proficiency as a Recoverable Dimension in Multilingual LLM Embeddings}},
  author = {Rodrigo Wilkens},
  booktitle = {Proceedings of the Joint Workshop on Readability and Text Simplification (READIxTSAR) @ LREC 2026},
  year = {2026},
  doi = {10.63317/3zrv8qescsdg},
  url = {https://doi.org/10.63317/3zrv8qescsdg}
}

References

36 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fei Dong and Yue Zhang. 2016. Automatic features for essay scoring–an empirical study. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1072–1077. find
  2. Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. 2016. Automatic text scoring using neural networks. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 715– 725.
  3. Muhammad Faseeh, Abdul Jaleel, Naeem Iqbal, Anwar Ghani, Akmalbek Abdusalomov, Asif Mehmood, and Young-Im Cho. 2024. Hybrid approach to automated essay scoring: Integrating deep learning embeddings with handcrafted linguistic features for improved accuracy. Mathematics, 12(21):3416. find
  4. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater® v. 2. The Journal of Technology, Learning and Assessment, 4(3). find
  5. Johannes Bjerva, Robert Östling, Maria Han Veiga, Jörg Tiedemann, and Isabelle Augenstein. 2019. What do language representations really represent? Computational Linguistics, 45(2):381– 389.
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Thomas François and Cédrick Fairon. 2012. An “ai readability” formula for french as a foreign language. In Proceedings of the 2012 joint conference on empirical methods in Natural Language Processing and comput…
  7. John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–…
  8. Zixuan Ke and Vincent Ng. 2019. Automated essay scoring: A survey of the state of the art. In IJCAI, volume 19, pages 6300–6308. find
  9. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483– 4499.
  10. Jesse Mu and Jacob Andreas. 2020. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153– 17163. find
  11. Council of Europe. 2020. Common European framework of reference for languages: Companion volume. Council of Europe. find
  12. Council of Europe. Council for Cultural Cooperation. Education Committee. Modern Languages Division. 2001. Common European framework of reference for languages: Learning, teaching, assessment. Cambridge University Press. find
  13. Xia Li, Huali Yang, Shengze Hu, Jing Geng, Keke Lin, and Yuhai Li. 2022. Enhanced hybrid neural network for automated essay scoring. Expert Systems, 39(10):e13068. find
  14. Ellis B Page. 1966. The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238–243. find
  15. Pei Yee Liew and Ian KT Tan. 2024. On automated essay grading using large language models. In Proceedings of the 2024 8th international conference on computer science and artificial intelligence, pages 204–211. find
  16. Isaac Persing, Alan Davis, and Vincent Ng. 2010. Modeling organization in student essays. In Proceedings of the 2010 conference on empirical methods in natural language processing, pages 229–239.
  17. Fengkai Liu and John SY Lee. 2023. Hybrid models for sentence readability assessment. In Proceedings of the 18th workshop on innovative use of nlp for building educational applications (bea 2023), pages 448–454. find
  18. Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-theoretic probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609–4622.
  19. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482. find
  20. Pedro Henrique Martins, João Alves, Patrick Fernandes, Nuno M Guerreiro, Ricardo Rei, Amin Farajian, Mateusz Klimaszewski, Duarte M Alves, José Pombal, Nicolas Boizard, et al. 2025a. Eurollm-9b: Technical report. arXiv preprint arXiv:2506.04079.
  21. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M Guerreiro, Ricardo Rei, Duarte M Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, et al. 2025b. Eurollm: Multilingual language models for europe. Procedia Computer Science, 255:53–62.
  22. Elijah Mayfield and Alan W Black. 2020. Should you fine-tune bert for automated essay scoring? In Proceedings of the fifteenth workshop on innovative use of NLP for building educational applications, pages 151–162. find
  23. Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050. find
  24. Pedro Uria Rodriguez, Amir Jafari, and Christopher M Ormerod. 2019. Language models and automated essay scoring. arXiv preprint arXiv:1909.09482. — [arXiv:1909.09482](https://arxiv.org/abs/1909.09482) find
  25. Kathrin Seßler, Maurice Fürstenberg, Babette Bühler, and Enkelejda Kasneci. 2025. Can ai grade your essays? a comparative analysis of large language models and teacher ratings in multidimensional essay scoring. In Proceedings of the 15th international learning analytics and knowledge conference, pag… find
  26. Mark D Shermis and Jill C Burstein. 2003. Automated essay scoring: A cross-disciplinary perspective. Routledge. find
  27. Kaveh Taghipour and Hwee Tou Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1882–1891.
  28. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 4593–4601.
  29. Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment research. In Proceedings of the thirteenth language resources and evaluation conference, pages 5366–5377. find
  30. Sowmya Vajjala and Taraka Rama. 2018. Experiments with universal cefr classification. In Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications, pages 147–153. find
  31. Rodrigo Wilkens, Patrick Watrin, Rémi Cardon, Alice Pintard, Isabelle Gribomont, and Thomas François. 2024. Exploring hybrid approaches to readability: experiments on the complementarity between linguistic features and transformers. In Findings of the Association for Computational Linguistics: EACL… find
  32. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, and A. M. Rush. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  33. Kevin P Yancey, Geoffrey Laflair, Anthony Verardi, and Jill Burstein. 2023. Rating short l2 essays on the cefr scale with gpt-4. In Proceedings of the 18th workshop on innovative use of NLP for building educational applications (BEA 2023), pages 576–584. find
  34. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading esol texts. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 180–189.
  35. Torsten Zesch, Michael Wojatzki, and Dirk Scholten-Akoun. 2015. Task-independent features for automated essay grading. In Proceedings of the tenth workshop on innovative use of NLP for building educational applications, pages 224–232. 7. find
  36. Language Resource References Joseph Marvin Imperial, Abdullah Barayan, Regina Stodden, Rodrigo Wilkens, Ricardo Muñoz Sánchez, Lingyun Gao, Melissa Torgbi, Dawn Knight, Gail Forey, Reka R Jablonkai, et al. 2025. Universalcefr: Enabling open multilingual research on language proficiency assessment. I… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wilkens-2026-language-proficiency-as-recoverable-dimension/