Elamur Research⌕
☰
home / papers / 2026 / Sakura at BEA 2026 Shared Task 1: What Makes Vocabulary Difficult?

Sakura at BEA 2026 Shared Task 1: What Makes Vocabulary Difficult?

Paper · 2026 · Adam Nohejl, Xuanxin Wu, Yusuke Ide et al.

Summary

The BEA 2026 Shared Task asked teams to predict how hard English words are to produce for learners whose L1 is Chinese, German or Spanish. The data come from the British Council's Knowledge-based Vocabulary Lists (KVL). Each item gives an L1 equivalent, an L1 context sentence, and a clue showing the first letter and word length. The target scores are GLMM intercepts, i.e. log-odds of a correct answer.

The Sakura team built two systems. The first fine-tunes LLMs and MLMs with a soft-target cross-entropy loss: each continuous score is split between its two neighbouring integer tokens, and predictions are read out as a probability-weighted mean. This avoids the precision lost when scores are rounded into discrete labels. This black-box model won the open track with r above 0.91. The second is an XGBoost regressor over interpretable features, analysed with SHAP. It reaches r above 0.77 and outperforms the fine-tuned encoder baseline in the closed track.

The analysis shows that KVL difficulty depends heavily on spelling and on how test items were written, not only on genuine word knowledge.

Key points

  • Soft-target fine-tuning: The training method is new for continuous-value prediction and beat both discretized LLM fine-tuning and MSE regression heads for MLMs. Performance kept improving with base model size up to 32B parameters, and the LLMs outperformed the smaller MLMs.
  • Explainable model features:
    • learner production frequency from the Lang-8 corpus;
    • reception frequency from TUBELEX and the spoken BNC;
    • CEFR level from the English Vocabulary Profile;
    • word length and Levenshtein-based L1 similarity;
    • LLM-derived judgements: spelling difficulty, lexical ambiguity and L1 calque status (from GPT-5.2), with ambiguity also rated by DeepSeek-V3.2.
  • Shared-task baseline: The organisers' baseline is a fine-tuned XLM-RoBERTa encoder (Skidmore et al., 2025).
  • Trickiness feature: This feature, the probability that GPT-4.1-mini answers an item wrongly, was allowed only in the open track. It revealed items where avoiding cognates leads respondents toward a different English word, for example inmediatamente suggesting "immediately" when the target is "instantly".
  • Resources and limitations: Code and an interactive explanation viewer are publicly released. The method has been tested only on KVL, and the explanations could not be validated because individual learner responses are not public.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nara Institute of Science and TechnologyIkoma · Japan46 works · 4,565 citations🎓National Tsing Hua UniversityHsinchu · Taiwan8 works · 834 citations🎓The University of OsakaOsaka · Japan15 works · 517 citations🔬RIKENWako · Japan26 works · 526 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

Cite

Cite as Adam Nohejl, Xuanxin Wu, Yusuke Ide, Maria Angelica Riera Machin, Yi-Ning Chang and Hitomi Yanaka. 2026. Sakura at BEA 2026 Shared Task 1: What Makes Vocabulary Difficult?. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.84

@inproceedings{nohejl2026sakura,
  title = {{Sakura at BEA 2026 Shared Task 1: What Makes Vocabulary Difficult?}},
  author = {Adam Nohejl, Xuanxin Wu, Yusuke Ide, Maria Angelica Riera Machin, Yi-Ning Chang and Hitomi Yanaka},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.84},
  url = {https://doi.org/10.18653/v1/2026.bea-1.84}
}

References

39 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dennis Aumiller and Michael Gertz. 2022. UniHD at TSAR-2022 shared task: Is compute all we need for lexical simplification? In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), pages 251–258, Abu Dhabi, United Arab Emirates (Virtual). Association for Com… find
  2. BNC Consortium. 2007. British National Corpus, XML edition. https://llds.ling-phil.ox.ac.uk/llds/xmlui/handle/20.500.14106/2554. ↗
  3. Annette Capel. 2012. Completing the English Vocabulary Profile: C1 and C2 vocabulary. English Profile Journal, 3:e1. find
  4. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, New York, NY, USA. Association for Computing Machinery. find
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  6. DeepSeek-AI. 2025. DeepSeek-V3 Technical Report. ArXiv preprint, arXiv:2412.19437v2 [cs.CL]. find
  7. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Fine-tuning of Quantized LLMs. Advances in Neural Information Processing Systems, 36:10088–10115. find
  8. Taisei Enomoto, Hwichan Kim, Tosho Hirasawa, Yoshinari Nagai, Ayako Sato, Kyotaro Nakajima, and Mamoru Komachi. 2024. TMU-HIT at MLSP 2024: How well can GPT-4 tackle multilingual lexical simplification? In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Application… find
  9. Mariano Felice and Lucy Skidmore. 2026. Findings of the BEA 2026 shared task on vocabulary difficulty prediction for English learners. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), San Diego, California. Association for Computational L…
  10. Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop.
  11. Yusuke Ide, Masato Mita, Adam Nohejl, Hiroki Ouchi, and Taro Watanabe. 2023. Japanese lexical complexity for non-native readers: A new dataset. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 477–487, Toronto, Canada. Association f… find
  12. John H.A.L. Jong, Mike Mayor, and Catherine Hayes. 2016. Developing global scale of English learning objectives aligned to the common European framework. Technical report. find
  13. Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Reso…
  14. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  15. Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, pages 4768–4777, Red Hook, NY, USA. Curran Associates Inc.
  16. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmBERT: A Modern Multilingual Encoder with Annealed Language Learning. ArXiv preprint, arXiv:2509.06888v1 [cs].
  17. Mistral AI. 2026. Ministral 3. arXiv:2601.08584v1 [cs.CL]. find
  18. ArXiv preprint, Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining revision log of language learning SNS for automated Japanese error correction of second language learners. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 147–… find
  19. Adam Nohejl, Akio Hayakawa, Yusuke Ide, and Taro Watanabe. 2024. Difficult for whom? a study of Japanese lexical complexity. In Proceedings of the Third Workshop on Text Simplification, Accessibility and Readability (TSAR 2024), pages 69–81, Miami, Florida, USA. Association for Computational Linguis… find
  20. Adam Nohejl, Akio Hayakawa, Yusuke Ide, and Taro Watanabe. 2025a. A Japanese Dataset and Efficient Multilingual LLM-Based Methods for Lexical Simplification and Lexical Complexity Prediction. Journal of Natural Language Processing, 32(4):1129–1188. find
  21. Adam Nohejl, Frederikus Hudi, Eunike Andriani Kardinata, Shintaro Ozaki, Maria Angelica Riera Machin, Hongyu Sun, Justin Vasselli, and Taro Watanabe. 2025b. Beyond film subtitles: Is YouTube the best approximation of spoken vocabulary? In Proceedings of the 31st International Conference on Computati… find
  22. OpenAI. 2024. GPT-4 Technical Report. ArXiv preprint, arXiv:2303.08774v6 [cs.CL].
  23. OpenAI. 2025. Update to GPT-5 System Card: GPT-5.2. Technical report. find
  24. Gustavo Paetzold and Lucia Specia. 2016. find
  25. SemEval 2016 task 11: Complex word identification. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 560–569, San Diego, California. Association for Computational Linguistics.
  26. Qwen Team. 2025. Qwen2.5 Technical Report. ArXiv preprint, arXiv:2412.15115v2 [cs.CL]. find
  27. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. In 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing @ NeurIPS 2019, volume arXiv:1910.01108. find
  28. Norbert Schmitt, Karen Dunn, Barry O’Sullivan, Laurence Anthony, and Benjamin Kremmel. 2021. Introducing Knowledge-based Vocabulary Lists (KVL). TESOL Journal, 12(4):e622. find
  29. Norbert Schmitt, Karen Dunn, Barry O’Sullivan, Laurence Anthony, and Benjamin Kremmel. 2024. Knowledge-Based Vocabulary Lists. British Council Monographs on Modern Language Testing. University of Toronto Press. find
  30. Graham G. Scott, Anne Keitel, Marc Becirspahic, Bo Yao, and Sara C. Sereno. 2019. The Glasgow Norms: Ratings of 5,500 words on nine scales. Behavior Research Methods, 51(3):1258–1270. find
  31. Seid Muhie Yimam, Chris Biemann, Shervin Malmasi, Gustavo Paetzold, Lucia Specia, Sanja Štajner, Anaïs Tack, and Marcos Zampieri. 2018. A report on the complex word identification shared task 2018. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applicatio… find
  32. A In the following, we list the full text of our prompt templates by feature for which they were used. The symbol ê stands for line breaks. Placeholders are printed in bold. The following common placeholders corresponding to the dataset items are used across multiple prompts: • {l1_name}: an L1, i.e… find
  33. The BEA 2024 shared task on the multilingual lexical simplification pipeline. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 571–589, Mexico City, Mexico. Association for Computational Linguistics. find
  34. Matthew Shardlow, Richard Evans, Gustavo Henrique Paetzold, and Marcos Zampieri. 2021. SemEval-2021 task 1: Lexical complexity prediction. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1–16, Online. Association for Computational Linguistics.
  35. Lucy Skidmore, Mariano Felice, and Karen Dunn. 2025. Transformer architectures for vocabulary test item difficulty prediction. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 160– 174, Vienna, Austria. Association for Computational… find
  36. Răzvan-Alexandru Smădu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop, and Mihaela-Claudia Cercel. 2024. Investigating large language models for complex word identification in multilingual and multidomain setups. In Proceedings of the 2024 Conference on Empirical Methods in Natural Languag… find
  37. Team GLM. 2024. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. ArXiv preprint, arXiv:2406.12793v2 [cs.CL].
  38. Prompt Templates • {l1_word}: an L1 word. • {l1_context}: an L1 context. • {en_word}: an English word. • {clue}: a letter-pattern clue derived from the English word (first letter + space-separated underscores), e.g., b _ _ _ for book. A.1 Fine-Tuning Basic template: Rate how difficult it is for lear… find
  39. [CLS] {l1_word} [SEP] {l1_context} [SEP] {clue} [SEP] {en_word} [SEP] find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nohejl-2026-sakura-bea-2026-shared-task/