Elamur Research⌕
☰
home / papers / 2026 / EduNLP at BEA 2026 Shared Task 1: Multi-Model Ensemble with Feature-Augmented Transformers for Vocabulary Difficulty Prediction

EduNLP at BEA 2026 Shared Task 1: Multi-Model Ensemble with Feature-Augmented Transformers for Vocabulary Difficulty Prediction

Paper · 2026 · Avinash Kumar Sharma

Summary

This system paper tackles the BEA 2026 shared task on estimating the difficulty of English vocabulary items for learners whose first language is Spanish, German or Mandarin. The targets are GLMM-calibrated scores from the Extended KVL Dataset for NLP. Most test words never appear in training, so models must generalize to unseen vocabulary. The author pairs hand-engineered signals (Zipf frequency, cognate similarity, length measures, POS flags) with fine-tuned xlm-roberta-base. The closed-track entry averages a per-L1 transformer with a LightGBM regressor. The open track adds two models: one trained jointly on all three L1s, and a variant that appends the 12 standardized features to the mean-pooled encoder output before an MLP head. The system beats the organizers' baselines in all six track–L1 settings, but a clear margin to the winning teams remains. The error analysis is useful in its own right. It identifies rare word senses and nominalized -ing forms as systematic weak spots. It also reports an overfitted Ridge blend as a cautionary negative result.

Key points

  • Data: 6,091 train, 677 dev and 748 test items per L1 (ES, DE, CN). About 82.5% of test words are absent from the train and dev sets.
  • Best test RMSEs were 1.058 (closed, CN) and 0.992 (open, CN). Gains over baselines were 0.081–0.134 (closed) and 0.042–0.095 (open). The winners Glite (closed) and Sakura (open) scored considerably lower.
  • In dev ablations, dropping frequency raised RMSE by 0.08–0.12 and dropping cognate features raised it by 0.04–0.07. LightGBM alone outperformed the transformer on roughly 27% of words.
  • Nominalized -ing words (e.g., "dining") had an RMSE of 1.735 versus 1.101 for other words, about 58% worse. Predictions were also compressed: the predicted standard deviation was about 81% of the true value for ES.
  • A Ridge blend with interaction features looked strong on dev but failed on test (DE 1.397, CN 1.421), worse than the baseline. The author attributes this to tuning on a single split without cross-validation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology MadrasChennai · India16 works · 2,499 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

Cite

Cite as Avinash Kumar Sharma. 2026. EduNLP at BEA 2026 Shared Task 1: Multi-Model Ensemble with Feature-Augmented Transformers for Vocabulary Difficulty Prediction. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.71

@inproceedings{sharma2026edunlp,
  title = {{EduNLP at BEA 2026 Shared Task 1: Multi-Model Ensemble with Feature-Augmented Transformers for Vocabulary Difficulty Prediction}},
  author = {Avinash Kumar Sharma},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.71},
  url = {https://doi.org/10.18653/v1/2026.bea-1.71}
}

References

8 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th annual meeting of the association fo… find
  2. Mariano Felice and Lucy Skidmore. 2026. Findings of the bea 2026 shared task on vocabulary difficulty prediction for english learners. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), San Diego, California. Association for Computational L…
  3. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30. find
  4. Norbert Schmitt, Karen Dunn, Barry O’Sullivan, Laurence Anthony, and Benjamin Kremmel. 2021. Introducing knowledge-based vocabulary lists (kvl). Tesol Journal, 12(4). find
  5. Norbert Schmitt, Karen Dunn, Barry O’Sullivan, Laurence Anthony, and Benjamin Kremmel. 2024. Knowledge-based vocabulary lists. University of Toronto Press. find
  6. Lucy Skidmore, Mariano Felice, and Karen Dunn. 2025. Transformer architectures for vocabulary test item difficulty prediction. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 160– 174, Vienna, Austria. Association for Computational… find
  7. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31. find
  8. Walter JB Van Heuven, Pawel Mandera, Emmanuel Keuleers, and Marc Brysbaert. 2014. Subtlex-uk: A new and improved word frequency database for british english. Quarterly journal of experimental psychology, 67(6):1176–1190. \#\# Acknowledgements We thank Dr. Shabana K M, Senior Project Scientist at the W… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sharma-2026-edunlp-bea-2026-shared-task/