Elamur Research⌕
☰
home / papers / 2026 / Data Asgardians at BEA 2026 Shared Task 1: A Hybrid Transformer–Feature Ensemble for L1-Aware English Vocabulary Difficulty Prediction

Data Asgardians at BEA 2026 Shared Task 1: A Hybrid Transformer–Feature Ensemble for L1-Aware English Vocabulary Difficulty Prediction

Paper · 2026 · Adrian Pineda, Sabur Butt and Héctor Ceballos Cancino

Summary

The authors address the BEA 2026 Shared Task 1, which asks systems to predict how hard English vocabulary items are for learners. The targets are GLMM-calibrated difficulty scores for speakers of Spanish, German and Mandarin Chinese. The data come from the British Council's Extended KVL Dataset, and the work follows the Closed Track rules. The team built 33 linguistic features from WordNet, the CMU Pronouncing Dictionary and PHOIBLE, covering:

  • phonology,
  • L1-specific sounds,
  • semantics and morphology,
  • cross-lingual properties.

They compared feature-only regressors, plain ("Solo") transformers, feature-augmented ("Hybrid") transformers and stacking. Their official XLM-RoBERTa-large runs beat the baseline for every L1. A later mDeBERTa-v3-base system, which combines four learners through Ridge stacking, lowered the mean test RMSE to 0.982. The study suggests that engineered linguistic cues help most when merged with transformer outputs at the prediction level. Feeding them directly into the network or using them alone works less well. This matters for scaling item calibration in adaptive testing without costly learner response collection.

Key points

  • Data: Extended KVL provides 6,091 train, 677 dev and 748 test items per L1. Difficulty scores for the same item correlate strongly across L1 groups (r≈0.78–0.85), but each group still shows its own variation.
  • Features: clue_ratio and word_frequency were the strongest single predictors. Even so, XGBoost and Ridge models using only the features failed to beat the XLM-R Base baseline (mean RMSE 1.218).
  • Official submission: the best XLM-R Large system, the Hybrid model, reached a mean test RMSE of 1.103. The best per-language results were 1.182 for ES (Solo), 1.117 for DE and 1.006 for CN (both Hybrid).
  • Post-submission ensemble: the mDeBERTa ensemble scored 1.037 (ES), 0.997 (DE) and 0.913 (CN). Its estimated, unofficial ranks are about 8/60, 11–12/57 and 15/52.
  • Effect of feature fusion: adding features directly helped XLM-R Large (1.103 vs. 1.122) but hurt mDeBERTa (1.150 vs. 1.110). Code is available on GitHub (bea2026_shared_task_DataAsgardians).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Tecnológico de Monterrey.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Adrian Pineda, Sabur Butt and Héctor Ceballos Cancino. 2026. Data Asgardians at BEA 2026 Shared Task 1: A Hybrid Transformer–Feature Ensemble for L1-Aware English Vocabulary Difficulty Prediction. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.82

@inproceedings{pineda2026data,
  title = {{Data Asgardians at BEA 2026 Shared Task 1: A Hybrid Transformer–Feature Ensemble for L1-Aware English Vocabulary Difficulty Prediction}},
  author = {Adrian Pineda, Sabur Butt and Héctor Ceballos Cancino},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.82},
  url = {https://doi.org/10.18653/v1/2026.bea-1.82}
}

References

13 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takuya Akiba, Shotaro Sano, Toshihiro Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2623–2631. ACM.
  2. R. Harald Baayen, Douglas J. Davidson, and Douglas M. Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4):390–412. ↗
  3. Carnegie Mellon University. 2014. The CMU pronouncing dictionary, v0.7b. find
  4. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. ACM. ↗
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451. Association for Computational Linguistics. find
  6. Mariano Felice and Lucy Skidmore. 2026. Findings of the BEA 2026 shared task on vocabulary difficulty prediction for English learners. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), San Diego, California. Association for Computational Linguistics.
  7. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In Proceedings of the 11th International Conference on Learning Representations.
  8. Batia Laufer and Zahava Goldstein. 2004. Testing vocabulary knowledge: Size, strength, and computer adaptiveness. Language Learning, 54(3):469–523. ↗
  9. George A. Miller. 1995. WordNet: A lexical database for English. Communications of the ACM, 38(11):39– 41. ↗
  10. Steven Moran and Daniel McCloy. 2019. PHOIBLE 2.0. find
  11. Matthew Shardlow, Richard Evans, Gustavo Henrique Paetzold, and Marcos Zampieri. 2021. SemEval-2021 task 1: Lexical complexity prediction. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1–16. Association for Computational Linguistics. ↗
  12. Lucy Skidmore, Mariano Felice, and Karen Dunn. 2025. Transformer architectures for vocabulary test item difficulty prediction. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 160– 174, Vienna, Austria. Association for Computational Linguistics. find
  13. David H. Wolpert. 1992. Stacked generalization. Neural Networks, 5(2):241–259. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pineda-2026-data-asgardians-bea-2026-shared/