Elamur Research⌕
☰
home / papers / 2026 / Glite at BEA 2026 Shared Task 1: Holistic Difficulty Models Dominate, Feature Engineering Closes the Gap in L1-Aware Vocabulary Difficulty Prediction

Glite at BEA 2026 Shared Task 1: Holistic Difficulty Models Dominate, Feature Engineering Closes the Gap in L1-Aware Vocabulary Difficulty Prediction

Paper · 2026 · Vassili Philippov, Dmitrii Andreev, Pavel Katunin et al.

Summary

The paper presents the Glite team's entry to the BEA 2026 shared task on predicting English vocabulary difficulty for learners with a specific first language: Spanish, German or Mandarin. The targets are psychometric GLMM scores from the British Council's Knowledge-based Vocabulary Lists. They are fitted on about 3.3M responses from more than 100,000 learners, who had to spell each target word exactly from an L1 cue.

For each L1, the team trains a separate CatBoost regressor on features chosen from 1,161 candidates. The candidates come from linguistic, psycholinguistic, dictionary and LLM sources, and they include out-of-fold predictions from fine-tuned encoder and decoder-LLM regressors. Recursive Feature Elimination with nested cross-validation selects the features. For the closed track, a column-by-column audit decided which features were allowed.

Across all three L1s, the system ranked first on the closed track and second on the open track. Its average RMSE was 29.9% (closed) and 35.9% (open) below the baseline. End-to-end "holistic" predictors, especially LoRA-tuned LLaMA-3.1-8B, carried most of the signal. This matters for scalable item calibration in adaptive testing and curriculum design, since such calibration otherwise needs costly pretesting.

Key points

  • Data: 7,516 English items, each in three L1 views (22,548 rows), split by item into train (6,091), dev (677) and test (748). The primary metric is RMSE.
  • Feature pool and selection: 129 feature sets in seven families, including target-word lexical, sense disambiguation, CEFR positioning, cue difficulty, virtual-learner simulation, L1→L2 transfer and holistic models. RFE uses 5×4 nested CV and keeps 29–150 features per L1.
  • Holistic models dominate: dropping the holistic family costs +0.102 RMSE, about ten times more than dropping any other family. LLaMA-3.1-8B LoRA alone reaches 0.831 RMSE. Mistral-7B (0.951) hurt the ensemble and was excluded.
  • Simple fusion is enough: a plain per-L1 CatBoost matched or beat Ridge stacking, and 6 of 63 models were enough to saturate the Ridge ensemble.
  • Closed-track compliance: 57 of 129 sets were judged eligible. Auditing at the column level rather than the set level kept 249 extra columns, including the RemBERT and XLM-R OOF predictions.
  • Leakage incident: four leaky feature sets inflated K-fold RMSE to 0.609. After they were removed, the clean result was 0.802.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SheffieldSheffield · United Kingdom113 works · 21,542 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

Cite

Cite as Vassili Philippov, Dmitrii Andreev, Pavel Katunin and Anton Nikolaev. 2026. Glite at BEA 2026 Shared Task 1: Holistic Difficulty Models Dominate, Feature Engineering Closes the Gap in L1-Aware Vocabulary Difficulty Prediction. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.77

@inproceedings{philippov2026glite,
  title = {{Glite at BEA 2026 Shared Task 1: Holistic Difficulty Models Dominate, Feature Engineering Closes the Gap in L1-Aware Vocabulary Difficulty Prediction}},
  author = {Vassili Philippov, Dmitrii Andreev, Pavel Katunin and Anton Nikolaev},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.77},
  url = {https://doi.org/10.18653/v1/2026.bea-1.77}
}

References

33 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. find
  2. The BEA 2019 shared task on grammatical error correction. In Proceedings of BEA 2019.
  3. Marc Brysbaert, Matthias Buchmeier, Markus Conrad, Arthur M. Jacobs, Jens Bölte, and Andrea Böhl. 2011. The word frequency effect: A review of recent developments and implications for the choice of frequency estimates in German. Experimental Psychology, 58(5):412–424. find
  4. Marc Brysbaert, Paweł Mandera, Samantha F. Mc-Cormick, and Emmanuel Keuleers. 2019. Word prevalence norms for 62,000 English lemmas. Behavior Research Methods. find
  5. Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known English word lemmas. Behavior Research Methods, 46(3):904–911. find
  6. Qing Cai and Marc Brysbaert. 2010. SUBTLEX-CH: Chinese word frequencies based on film subtitles. PLoS ONE, 5(6):e10729. find
  7. Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking embedding coupling in pre-trained language models. In Proceedings of ICLR.
  8. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In Proceedings of ICLR. find
  9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL. find
  10. Fernando Cuetos, María Glez-Nosti, Analia Barbón, and Marc Brysbaert. 2011. SUBTLEX-ESP: Spanish word frequencies based on film subtitles. Psicológica, 32:133–143. find
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The LLaMA 3 herd of models. arXiv:2407.21783.
  12. Mariano Felice and Lucy Skidmore. 2026. Findings of the BEA 2026 shared task on vocabulary difficulty prediction for English learners. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), San Diego, California. Association for Computational L…
  13. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, and 1 others. 2023. Mistral 7B. arXiv:2310.06825. find
  14. Masato Kikuchi, Masatsugu Ono, Toshioki Soga, Tetsu Tanabe, and Tadachika Ozono. 2025. CEFR-annotated WordNet: LLM-based proficiency-guided semantic database for language learning. arXiv preprint arXiv:2510.18466. find
  15. Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie-Yan Liu. 2021. R-Drop: Regularized dropout for neural networks. In Proceedings of NeurIPS. find
  16. Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of LREC.
  17. Gustavo Paetzold and Lucia Specia. 2016. find
  18. SemEval 2016 task 11: Complex word identification. In Proceedings of SemEval-2016.
  19. Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. 2018. CatBoost: unbiased boosting with categorical features. In Proceedings of NeurIPS. find
  20. Claudia H. Sánchez-Gutiérrez, Hugo Mailhot, S. Hélène Deacon, and Maximiliano A. Wilson. 2018. MorphoLex: A derivational morphological database for english. In Behavior Research Methods. find
  21. Norbert Schmitt, Karen Dunn, Barry O’Sullivan, Laurence Anthony, and Benjamin Kremmel. 2024. Knowledge-based Vocabulary Lists, volume 5 of British Council Monographs on Modern Language Testing. University of Toronto Press. find
  22. Christina L. Gagné, Thomas L. Spalding, and Daniel Schmidtke. 2019. LADEC: The large database of english compounds. Behavior Research Methods. find
  23. Matthew Shardlow, Richard Evans, Gustavo Paetzold, and Marcos Zampieri. 2021. SemEval-2021 task 1: Lexical complexity prediction. In Proceedings of SemEval-2021.
  24. Chuanji Gao, Svetlana V. Shinkareva, and Rutvik H. Desai. 2023. SCOPE: The South Carolina psycholinguistic metabase. Behavior Research Methods, 55:2853–2884. find
  25. Matthew Shardlow, Sanja Štajner, Kai North, Tharindu Ranasinghe, and Marcos Zampieri. 2024. find
  26. The BEA 2024 shared task on the multilingual lexical simplification pipeline. In Proceedings of BEA 2024. find
  27. Sebastian Gombert, Lukas Menzel, Daniele Di Mitri, and Hendrik Drachsler. 2024. Predicting item difficulty and item response time with scalar-mixed transformer encoder models and rational network regression heads. In Proceedings of BEA 2024. find
  28. Yukio Tono. 2019. CEFR-J wordlist version 1.5. Tokyo University of Foreign Studies, Tono Laboratory. http://www.cefr-j.org/download.html. ↗
  29. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of ACL. find
  30. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In Proceedings of ICLR.
  31. Jue Wang and Baoguo Chen. 2020. A database of Chinese-English bilingual speakers: Ratings of the age of acquisition and familiarity. Frontiers in Psychology, 11:554785. find
  32. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, and 1 others. 2024. Qwen2.5: A party of foundation models. arXiv:2412.15115. find
  33. Seid Muhie Yimam, Chris Biemann, Sanja Štajner, Sian Gooding, and Matthew Shardlow. 2018. A report on the complex word identification shared task 2018. In Proceedings of BEA 2018. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/philippov-2026-glite-bea-2026-shared-task/