Elamur Research⌕
☰
home / papers / 2026 / SAAKTH at BEA 2026 Shared Task 1: L1-Aware English Vocabulary Difficulty Prediction with Hybrid Transformer and Psycholinguistic Features

SAAKTH at BEA 2026 Shared Task 1: L1-Aware English Vocabulary Difficulty Prediction with Hybrid Transformer and Psycholinguistic Features

Paper · 2026 · Karthik Mattu, Adit Dhall, Arshad Naguru et al.

Summary

The paper describes team SAAKTH's submission to the closed track of the BEA 2026 shared task, which focuses on predicting vocabulary difficulty. The task is to estimate how hard an English word is for learners whose first language is Spanish, German, or Mandarin. Difficulty partly depends on the learner's L1: cross-L1 score correlations in the training data reach only r = 0.63–0.68. For that reason, the authors train a separate model for each L1 group.

Each model concatenates the first-token embedding from a fine-tuned XLM-RoBERTa-large with a small vector of handcrafted features. These include frequency, age of acquisition, concreteness, CEFR level, WordNet and orthographic cues, cognate similarity for the European languages, and character stroke count for Mandarin. A shallow MLP processes the combined vector. Predictions from five seeds are averaged and then linearly blended with a feature-only XGBoost model.

On the development set, this setup cuts RMSE by roughly 20–26% relative to the organizers' XLM-RoBERTa-base baseline. The results suggest that targeted L1-specific features help when data is limited. They also show weak transfer between typologically distant L1s.

Key points

  • Data: each L1 (es, de, cn) has 6,091 training and 677 dev items, derived from the British Council's KVL. Targets are GLMM-based difficulty scores.
  • Dev RMSE: 0.997 (es), 1.002 (de), and 0.932 (cn), compared with baseline scores of 1.357, 1.328, and 1.175. Test RMSE was 1.045, 0.994, and 0.900.
  • Feature sets: 19 (es), 20 (de), and 16 (cn) features, chosen from 21 candidates. Training used dual learning rates (1e-5 for the encoder, 1e-4 for the MLP head). The XGBoost blend weight α was 0.8 for es/de and 0.9 for cn.
  • Feature importance: SHAP analysis ranks age of acquisition and SUBTLEX-US frequency highest for es/de. For cn, stroke count (Unihan) dominates with mean |SHAP| = 0.078. In ablations, removing cognate features unexpectedly helped German, and removing context features helped Spanish.
  • Cross-L1 transfer: Spanish and German models transfer to each other with modest losses (+0.02–0.14 RMSE), while Mandarin-trained models lose about +0.52. Code is released on GitHub (aditdhall/bea2026-vocab-difficulty).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Rochester Institute of TechnologyRochester · United States15 works · 5,233 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksPsycholinguistics & language acquisition · 102 works

Cite

Cite as Karthik Mattu, Adit Dhall, Arshad Naguru, Shubh Sehgal, Thejas Gowda and Hakyung Sung. 2026. SAAKTH at BEA 2026 Shared Task 1: L1-Aware English Vocabulary Difficulty Prediction with Hybrid Transformer and Psycholinguistic Features. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.69

@inproceedings{mattu2026saakth,
  title = {{SAAKTH at BEA 2026 Shared Task 1: L1-Aware English Vocabulary Difficulty Prediction with Hybrid Transformer and Psycholinguistic Features}},
  author = {Karthik Mattu, Adit Dhall, Arshad Naguru, Shubh Sehgal, Thejas Gowda and Hakyung Sung},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.69},
  url = {https://doi.org/10.18653/v1/2026.bea-1.69}
}

References

33 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marc Brysbaert and Boris New. 2009. Moving beyond Kučera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41(4):977– 990. ↗
  2. Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known English word lemmas. Behavior Research Methods, 46(3):904–911. ↗
  3. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of KDD. ↗
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL, pages 8440–8451. find
  5. Council of Europe. 2001. Common European framework of reference for languages. find
  6. Scott Crossley, Kristopher Kyle, and Thomas Salsbury. 2016. A usage-based investigation of l2 lexical acquisition: The role of input and output. The Modern Language Journal, 100(3):702–715. ↗
  7. Ton Dijkstra, Koji Miwa, Bianca Brummelhuis, Maya Sappelli, and Harald Baayen. 2010. How cross-language similarity and task demands affect cognate recognition. Journal of Memory and Language, 62(3):284–301. ↗
  8. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv:2002.06305. ↗
  9. Mariano Felice and Lucy Skidmore. 2026. Findings of the bea 2026 shared task on vocabulary difficulty prediction for english learners. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), San Diego, California. Association for Computational Linguistics.
  10. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339. ↗
  11. Minkyung Kim, Scott A Crossley, and Kristopher Kyle. 2018. Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. The Modern Language Journal, 102(1):120–141. ↗
  12. Victor Kuperman, Hans Stadthagen-Gonzalez, and Marc Brysbaert. 2012. Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods, 44(4):978–990. ↗
  13. Batia Laufer. 1997. What’s in a word that makes it hard or easy? In Vocabulary: Description, Acquisition and Pedagogy, pages 140–155. find
  14. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707–710. find
  15. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR). find
  16. George A. Miller. 1995. Wordnet: A lexical database for english. Communications of the ACM, 38(11):39– 41. ↗
  17. Paul Nation. 2001. Learning Vocabulary in Another Language. Cambridge University Press. ↗
  18. Terence Odlin. 2003. Cross-linguistic influence. The handbook of second language acquisition, pages 436– 486. ↗
  19. Gustavo H. Paetzold and Lucia Specia. 2016. Semeval 2016 task 11: Complex word identification. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 560–569, San Diego, California. Association for Computational Linguistics.
  20. Matthew Shardlow and 1 others. 2024. Bea 2024 shared task on the multilingual lexical simplification pipeline. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Mexico City, Mexico. Association for Computational Linguistics. find
  21. Hua Shu, Xi Chen, Richard C. Anderson, Ningning Wu, and Yan Xuan. 2003. Properties of school chinese: Implications for learning to read. Child Development, 74(1):27–47. ↗
  22. Lucy Skidmore, Mariano Felice, and Karen J. Dunn. 2025. Transformer architectures for vocabulary test item difficulty prediction. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 160– 174, Location TBD. Association for Computational Linguistics. find
  23. Hakyung Sung, Mikyung Kim Wolf, Michael Suhan, and Kristopher Kyle. 2025. Lexical richness in young english learners’ writing: A focus on opinion and listen-write task types. Assessing Writing, 66:100975. ↗
  24. Unicode Consortium. 2024. The unicode standard, version 16.0: Unihan database. https://www.unicode.org/standard/standard.html. ↗
  25. Adam Paszke, Sam Gross, Francesco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS).
  26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Timothee Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.
  27. Håkan Ringbom. 2007. Cross-linguistic Similarity in Foreign Language Learning. Multilingual Matters. ↗
  28. David H. Wolpert. 1992. Stacked generalization. Neural Networks, 5(2):241–259. find
  29. Job Schepens, Ton Dijkstra, and Franc Grootjen. 2012. Distributions of cognates in europe as based on levenshtein distance. Bilingualism: Language and Cognition, 15(1):157–166. ↗
  30. Seid Muhie Yimam, Chris Biemann, Shervin Malmasi, Gustavo Henrique Paetzold, Lucia Specia, Svenja Ser, Lea A. Deleris, and Marcos Zampieri. 2018. A report on the complex word identification shared task 2018. In Proceedings of the 13th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2018), pages 66–78, New Orleans, Louisiana. Association for Computational Linguistics. find
  31. Norbert Schmitt, Karen Dunn, Barry O’Sullivan, Laurence Anthony, and Benjamin Kremmel. 2024. Knowledge-based Vocabulary Lists. University of Toronto Press, Toronto. find
  32. Burr Settles, Geoffrey T. LaFlair, and Masato Hagiwara. 2020. Machine learning–driven language assessment. Transactions of the Association for Computational Linguistics, 8:247–263. ↗
  33. Matthew Shardlow, Richard Evans, Gustavo Henrique Paetzold, and Marcos Zampieri. 2021. Semeval-2021 task 1: Lexical complexity prediction. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1–16, Online. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mattu-2026-saakth-bea-2026-shared-task/