Elamur Research⌕
☰
home / papers / 2026 / RETUYT-INCO at BEA 2026 Shared Task 1: Feature-Enriched mDeBERTa for Word Difficulty Prediction

RETUYT-INCO at BEA 2026 Shared Task 1: Feature-Enriched mDeBERTa for Word Difficulty Prediction

Paper · 2026 · Santiago Robaina, Aiala Rosá and Luis Chiruzzo

Summary

This paper presents the RETUYT-INCO team's submission to the 2026 BEA shared task, which focuses on predicting how difficult vocabulary items are. The task asks systems to estimate a continuous GLMM psychometric difficulty score for English words presented to learners whose L1 is Spanish, German or Mandarin. The authors stayed in the closed track and limited themselves to models that run on CPU or a single modest GPU.

They built two systems:

  • XGBoost regressor (all three L1s): uses handcrafted features, including word length, POS, Zipf frequency, affix indicators, cross-lingual string similarity, and a LaBSE cosine between the L1 cue and the English word.
  • mDeBERTa ensemble (Spanish only): fine-tunes mdeberta-v3-base with selected features written as text tokens at the start of the input, then averages three seeds.

The Spanish ensemble reached 1.094 test RMSE. This is 13.0% below the XLM-RoBERTa-base baseline and placed the team 9th of 21. Feature-only XGBoost slightly outperformed the baseline on average dev RMSE. This suggests that cheap, interpretable models remain competitive for lexical difficulty prediction when compute is scarce.

Key points

  • Data and evaluation: Each L1 has 6,091 training and 677 dev items. Targets range from roughly −6 to +5, and lower scores mean harder words. RMSE is the main metric and Pearson the secondary one.
  • LaBSE cosine feature (embed_cosine): It gave the biggest single-feature gain in the incremental ablation, cutting average dev RMSE by 0.091 (1.381 → 1.290). Mandarin benefited most (1.257 → 1.181). The authors note this result depends on the order in which features were added.
  • XGBoost results: Average dev RMSE was 1.273, against 1.287 for the baseline. Test results were mixed relative to the baseline:
    • Mandarin: 1.106 vs. 1.140 (better).
    • German: 1.260 vs. 1.258 (roughly equal).
    • Spanish: 1.323 vs. 1.257 (worse), even though it beat the baseline on dev.
  • mDeBERTa ensemble: Each model has about 276M parameters, so the three-seed ensemble loads about 828M at inference. Seeds were 10, 42 and 123, and training ran on a Colab T4. It achieved 1.103 dev RMSE on Spanish and 0.843 test Pearson. The final model was retrained on train+dev before predicting the test set.
  • Hyperparameter tuning: A random search over XGBoost settings changed dev RMSE by only about 0.001, so feature engineering drove most of the gains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universidad de la República de UruguayMontevideo · Uruguay9 works · 450 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

Cite

Cite as Santiago Robaina, Aiala Rosá and Luis Chiruzzo. 2026. RETUYT-INCO at BEA 2026 Shared Task 1: Feature-Enriched mDeBERTa for Word Difficulty Prediction. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.79

@inproceedings{robaina2026retuyt,
  title = {{RETUYT-INCO at BEA 2026 Shared Task 1: Feature-Enriched mDeBERTa for Word Difficulty Prediction}},
  author = {Santiago Robaina, Aiala Rosá and Luis Chiruzzo},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.79},
  url = {https://doi.org/10.18653/v1/2026.bea-1.79}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785– 794. find
  2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. vocabulary test item responses. Research… find
  3. Mariano Felice and Lucy Skidmore. 2026. Findings of the BEA 2026 Shared Task on Vocabulary Difficulty Prediction for English Learners. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications. To appear.
  4. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891. Association for Computational Linguistics.
  5. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543.
  6. Batia Laufer and Zahava Goldstein. 2004. Testing vocabulary knowledge: Size, strength, and computer adaptiveness. Language Learning, 54(3):399–436. find
  7. Paul De Boeck. 2008. Random item IRT models. Psychometrika, 73(4):533–559. find
  8. Gustavo Paetzold and Lucia Specia. 2016. find
  9. SemEval 2016 task 11: Complex word identification. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 560–569, San Diego, California. Association for Computational Linguistics.
  10. Karen J. Dunn. 2024. Random-item Rasch models and explanatory extensions: A worked example using L2 find
  11. Norbert Schmitt, Karen Dunn, Barry O’Sullivan, Laurence Anthony, and Benjamin Kremmel. 2021. Introducing knowledge-based vocabulary lists (KVL). TESOL Journal, 12(4). find
  12. Norbert Schmitt, Karen Dunn, Barry O’Sullivan, Laurence Anthony, and Benjamin Kremmel. 2024. Knowledge-based Vocabulary Lists. University of Toronto Press, Toronto. find
  13. Matthew Shardlow, Fernando Alva-Manchego, Riza Batista-Navarro, Stefan Bott, Saul Calderon Ramirez, Rémi Cardon, Thomas François, Akio Hayakawa, Andrea Horbach, Anna Hülsing, Yusuke Ide, Joseph Marvin Imperial, Adam Nohejl, Kai North, Laura Occhipinti, Nelson Peréz Rojas, Nishat Raihan, Tharindu Ran… find
  14. The BEA 2024 shared task on the multilingual lexical simplification pipeline. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 571–589, Mexico City, Mexico. Association for Computational Linguistics. find
  15. Matthew Shardlow, Richard Evans, Gustavo Henrique Paetzold, and Marcos Zampieri. 2021. SemEval-2021 task 1: Lexical complexity prediction. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1–16, Online. Association for Computational Linguistics.
  16. Lucy Skidmore, Mariano Felice, and Karen Dunn. 2025. Transformer architectures for vocabulary test prediction. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 160–174, Vienna, Austria. Association for Computational Linguistics. find
  17. Robyn Speer. 2022. rspeer/wordfreq: v3.0. find
  18. William E. Winkler. 1990. String comparator metrics and enhanced decision rules in the Fellegi-Sunter model of record linkage. In Proceedings of the Section on Survey Research Methods, American Statistical Association. find
  19. Seid Muhie Yimam, Chris Biemann, Shervin Malmasi, Gustavo Paetzold, Lucia Specia, Sanja Štajner, Anaïs Tack, and Marcos Zampieri. 2018. A report on the complex word identification shared task 2018. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applicatio… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/robaina-2026-retuyt-inco-bea-2026-shared/