Elamur Research⌕
☰
home / papers / 2026 / SATLab at BEA 2026 Shared Task 1: Predicting the Difficulty of English Words for Three L1 Learners Using Primarily Psycholinguistic Features

SATLab at BEA 2026 Shared Task 1: Predicting the Difficulty of English Words for Three L1 Learners Using Primarily Psycholinguistic Features

Paper · 2026 · Yves Bestgen

Summary

This paper presents the system SATLab submitted to a BEA 2026 shared task focused on how difficult vocabulary items are. The task was to predict how hard English words are for learners whose first language is Spanish, German or Mandarin. The data comes from the Extended Knowledge-based Vocabulary Lists Dataset: 7,516 lemmas whose difficulty was estimated psychometrically from a translation test taken by over 100,000 learners.

Instead of relying on embeddings, the author built a large set of hand-picked features:

  • frequency counts from social media, web and classic corpora,
  • psycholinguistic norms (Glasgow, ELP),
  • learner-oriented word lists (Oxford 3000, BKL2, LexCom),
  • orthographic and phonetic similarity between the English word and its L1 translation.

These features feed a LightGBM regressor. One variant uses only these features (Nonly). The other adds the organisers' transformer baseline prediction as one more input (Dev).

Both variants beat the baseline on RMSE but fell well short of the best team. The study matters because it shows how far interpretable lexical resources can go on their own. It also shows they still add signal when combined with a neural predictor.

Key points

  • The data splits are 6,091 learning, 677 development and 748 test items. Systems were scored by RMSE in a closed track (no LLMs, no mixing of L1s) and an open track.

  • Mean RMSE across tracks and L1s:

    • Dev version: 1.019
    • Baseline: 1.176
    • Winning team: 0.790

    In the closed track, Nonly scored 1.136 against the baseline's 1.218, and Dev scored 1.025.

  • By LightGBM gain, the strongest feature groups were Sim12 (Levenshtein + Soundex similarity, German/Spanish only), BKL2 and Oxford. Twitter and Facebook frequencies beat the other frequency lists.

  • In Spearman correlation, Google Ngram, USENET and BNC frequencies never reached the top ten. LexCom ranked near the top in correlation for every L1, yet contributed little gain.

  • Error analysis covered the 20 worst items per L1:

    • Over 90% of these items were specific to one L1.
    • About 90% were underestimates of difficulty, with errors approaching 5 on a scale of roughly −6 to +5.
    • Many involved a rare use of a common word in an unusual part of speech, such as "clear" as an adverb or "very" as an emphatic adjective.

Interesting ideas

  • I1. Reading gain-based importance next to raw rank correlation separates two kinds of feature: those that carry unique information (Sim12 has a modest correlation but high gain) and those that are strong but redundant (LexCom has a high correlation but low gain).
  • I2. Feeding a black-box model's prediction into a transparent tree model as just one more input lets you measure how much the hand-built features add on top of it.
  • I3. The worst errors came from familiar word forms used in a rare grammatical category, so frequency should be counted per form and part of speech, not per surface form alone.
  • I4. Hyperparameters were scored by the average over 100 bootstrap resamples of a small dev set, rather than on the single dev split.
  • I5. Cheap cross-lingual similarity (edit distance plus Soundex) between a word and its translation served as a strong proxy for cognate status.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When evaluating which signals improve a learned component, report both views, so a redundant but well-correlated feature is not mistaken for an indispensable one.
  • I2 (learn): A neural parser's (gparse) confidence could be one feature in an explainable arbiter over witness parsers, showing where symbolic cues still correct it.
  • I3 (tokens): Store lemma+POS frequencies in crate en and flag low-probability category uses (adverbial "clear", adjectival "very") for extra tagger scrutiny.
  • I4 (learn): Bootstrapped dev scoring would make model selection on small treebank dev sets less sensitive to a few lucky items.
  • I5 (back): During translation through the meta-world, a similar score could flag likely cognates and false friends for extra checking.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓KU LeuvenLeuven · Belgium42 works · 5,839 citations🎓UCLouvainLouvain-la-Neuve · Belgium20 works · 3,741 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksPsycholinguistics & language acquisition · 102 works

Cite

Cite as Yves Bestgen. 2026. SATLab at BEA 2026 Shared Task 1: Predicting the Difficulty of English Words for Three L1 Learners Using Primarily Psycholinguistic Features. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.66

@inproceedings{bestgen2026satlab,
  title = {{SATLab at BEA 2026 Shared Task 1: Predicting the Difficulty of English Words for Three L1 Learners Using Primarily Psycholinguistic Features}},
  author = {Yves Bestgen},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.66},
  url = {https://doi.org/10.18653/v1/2026.bea-1.66}
}

References

27 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David A. Balota, Melvin J. Yap, Keith A. Hutchison, Michael J. Cortese, Brett Kessler, Bjorn Loftis, James H. Neely, Douglas L. Nelson, Greg B. Simpson, and Rebecca Treiman. 2007. The English lexicon project. Behavior Research Methods, 39:445– 459. ↗
  2. Yves Bestgen. 2021a. LAST at CMCL 2021 shared task: Predicting gaze data during reading with a gradient boosting decision tree approach. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 90–96, Online. Association for Computational Linguistics. find
  3. Yves Bestgen. 2021b. LAST at SemEval-2021 Task 1: improving multi-word complexity prediction using bigram association measures. In Proceedings of SemEval-2021. find
  4. Shin’ichiro Ishikawa. 2023. The ICNALE Guide: An Introduction to a Learner Corpus Study on Asian Learners’ L2 English. Routledge. find
  5. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A highly efficient gradient boosting decision tree. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 3146–3154. Curran Associates, Inc. find
  6. Rebecca E. Knoph, Joshua F. Lawrence, and David J. Francis. 2024. The dimensionality of lexical features in general, academic, and disciplinary vocabulary. Scientific Studies of Reading, 28(4):319–342. ↗
  7. Kevin Lund and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28(2):203–208. ↗
  8. Mounica Maddela and Wei Xu. 2018. A word-complexity lexicon and a neural readability ranking model for lexical simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3749–3760, Brussels, Belgium. Association for Computational Linguistics. find
  9. Yves Bestgen. 2021c. A simple language-agnostic yet strong baseline system for hate speech and offensive content identification. In Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR Workshop Proceedings. CEUR-WS.org. find
  10. Alejandro Mosquera. 2021. Alejandro mosquera at SemEval-2021 task 1: Exploring sentence and word features for lexical complexity prediction. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 554–559, Online. Association for Computational Linguistics. find
  11. Marc Brysbaert, Emmanuel Keuleers, and Paweł Mandera. 2020. Which words do English non-native speakers know? New supernational levels based on yes/no decision. Second Language Research, 36(4):395–417. find
  12. Agnieszka Otwinowska and Jakub M. Szewczyk. 2019. The more similar the better? Factors in learning cognates, false cognates and non-cognate words. International Journal of Bilingual Education and Bilingualism, 22(8):974–991. find
  13. Brent Culligan. 2015. A comparison of three test formats to assess word difficulty. Language Testing, 32(4):503–520. ↗
  14. Norbert Schmitt, Karen Dunn, Barry O’Sullivan, Laurence Anthony, and Benjamin Kremmel. 2021. Introducing Knowledge-based Vocabulary Lists (KVL). TESOL Journal, 12(4):e622. ↗
  15. Mariano Felice and Lucy Skidmore. 2026. Findings of the BEA 2026 shared task on vocabulary difficulty prediction for English learners. In Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), San Diego, California. Association for Computational Linguistics.
  16. Norbert Schmitt, Karen Dunn, Barry O’Sullivan, Laurence Anthony, and Benjamin Kremmel. 2024. Knowledge-based Vocabulary Lists. University of Toronto Press. find
  17. Norbert Schmitt and Diane Schmitt. 2020. Vocabulary in Language Teaching. Cambridge University Press. ↗
  18. Steve Graham, Karen R Harris, and Connie Loynachan. 1993. The basic spelling vocabulary list. The Journal of Educational Research, 86(6):363–368. ↗
  19. Amac Herdagdelen and Marco Marelli. 2017. Social media and language processing: How Facebook and Twitter provide the best frequency estimates for studying word recognition. Cognitive Science, 41:976–995. ↗
  20. Graham G. Scott, Anne Keitel, Marc Becirspahic, Bo Yao, and Sara C. Sereno. 2019. The Glasgow norms: Ratings of 5,500 words on nine scales. Behavior Research Methods, 51:1258–1270. ↗
  21. Burr Settles, Geoffrey T. LaFlair, and Masato Hagiwara. 2020. Machine learning–driven language assessment. Transactions of the Association for Computational Linguistics, 8:247–263. ↗
  22. Cyrus Shaoul and Westbury Chris. 2006. USENET orthographic frequencies for 111,627 English words. find
  23. Matthew Shardlow, Richard Evans, Gustavo Henrique Paetzold, and Marcos Zampieri. 2021. SemEval-2021 task 1: Lexical complexity prediction. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1–16, Online. Association for Computational Linguistics. ↗
  24. Lucy Skidmore, Mariano Felice, and Karen Dunn. 2025. Transformer architectures for vocabulary test item difficulty prediction. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 160– 174, Vienna, Austria. Association for Computational Linguistics. find
  25. Victoria Yaneva, Kai North, Peter Baldwin, Le An Ha, Saed Rezayi, Yiyun Zhou, Sagnik Ray Choudhury, Polina Harik, and Brian Clauser. 2024. Findings from the first shared task on automated prediction of difficulty and response time for multiple-choice questions. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 470–482, Mexico City, Mexico. Association for Computational Linguistics. find
  26. Helen Yannakoudakis and Ted Briscoe. 2012. Modeling coherence in ESOL learner texts. In Proceedings of the 7th Workshop on the Innovative Use of NLP for Building Educational Applications, pages 33–43. find
  27. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180–189.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bestgen-2026-satlab-bea-2026-shared-task/