Elamur Research⌕
☰
home / papers / 2026 / SurreyCTS at BEA 2026 Shared Task 1: Semantic Funnelling and Entropy-based Multilingual Lexical Difficulty Prediction

SurreyCTS at BEA 2026 Shared Task 1: Semantic Funnelling and Entropy-based Multilingual Lexical Difficulty Prediction

Paper · 2026 · Georgina Willoughby, Jordan Painter, Diptesh Kanojia et al.

Summary

The University of Surrey team entered the open track of the BEA 2026 shared task. The task asks systems to predict how hard English vocabulary items are for learners whose first language is Spanish, German or Chinese. The targets are continuous GLMM (Rasch-type) difficulty estimates from the Extended KVL Dataset, which is based on British Council vocabulary lists and roughly 3.3 million responses from over 100,000 learners.

The authors started from multilingual encoders (XLM-R, mDeBERTa, COMET, RemBERT) and built a hybrid regressor. It mean-pools target-word states from several encoder layers ("semantic funnelling") and adds string-similarity, subword-ratio, attention-entropy and language-ID signals. These go into a deep MLP head trained with Huber loss. A redesigned, difficulty-stratified validation split that pools all three L1s proved essential for choosing models.

The final entry was a five-model ensemble weighted by inverse squared RMSE. It beat the organisers' open baseline for every L1 and ranked fifth among open-track teams. The work suggests that a few well-chosen linguistic features still help even with strong pretrained encoders.

Key points

  • Best single models (internal 500-item validation set):
    • prod-H (RemBERT + funnelling + language-ID prefix) had the lowest RMSE, 0.8122.
    • prod-G (COMET with minimal features) had the highest Pearson, 0.8968.
    • Vanilla RemBERT reached RMSE 0.8415.
  • Official test RMSE, ensemble vs. open baseline:
    • Spanish: 1.034 vs. 1.198
    • German: 0.945 vs. 1.166
    • Chinese: 0.861 vs. 1.034
  • Validation split redesign:
    • Encoder-only models stalled near RMSE 1.00.
    • The official 2,031-item dev set did not separate configurations reliably.
    • The authors split labelled data into 50 GLMM quantile buckets and took the 10 top-scoring items from each, giving a 500-item dev set biased toward harder items and 19,804 training items.
    • Mixing L1s in training forced entry into the open track.
  • Ablation findings:
    • Targeted feature additions beat the full feature bundle.
    • TSA was the most useful entropy feature.
    • COMET worked best with minimal features, while RemBERT gained from the additions. The authors suggest COMET's smaller embedding size (d = 512 vs. 1152) may explain this.
  • L1 differences:
    • For Spanish (−0.24, −0.27) and German (−0.34, −0.35) learners, normalised Levenshtein and Jaccard show an inverse relationship with difficulty.
    • For Chinese learners these correlations are near zero, even after pinyin romanisation.
    • Subword ratio behaves consistently across all three L1s.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SurreyGuildford · United Kingdom21 works · 677 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Georgina Willoughby, Jordan Painter, Diptesh Kanojia, Emily Wells and Constantin Orasan. 2026. SurreyCTS at BEA 2026 Shared Task 1: Semantic Funnelling and Entropy-based Multilingual Lexical Difficulty Prediction. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.70

@inproceedings{willoughby2026surreycts,
  title = {{SurreyCTS at BEA 2026 Shared Task 1: Semantic Funnelling and Entropy-based Multilingual Lexical Difficulty Prediction}},
  author = {Georgina Willoughby, Jordan Painter, Diptesh Kanojia, Emily Wells and Constantin Orasan},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.70},
  url = {https://doi.org/10.18653/v1/2026.bea-1.70}
}

References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking embedding coupling in pre-trained language models. In International Conference on Learning Representations.
  2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  3. Paul De Boeck. 2008. Random item irt models. Psychometrika, 73(4):533–559. find
  4. Karen J. Dunn. 2024. Random-item rasch models and explanatory extensions: A worked example using l2 vocabulary test item responses. Research Methods in Applied Linguistics, 3(3):100143. find
  5. Namrata Bhalchandra Patil Gurav, Akashdeep Ranu, Archchana Sindhujan, and Diptesh Kanojia. 2026. Domain-specific quality estimation for machine translation in low-resource scenarios. In Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026), pages 630– 650, R… find
  6. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with disentangled attention. In International Conference on Learning Representations.
  7. Peter J. Huber. 1964. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73–101. find
  8. Diptesh Kanojia, Malhar Kulkarni, Pushpak Bhattacharyya, and Gholamreza Haffari. 2020. Challenge dataset of cognates and false friend pairs from Indian languages. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3096–3102, Marseille, France. European Language Resourc… find
  9. Diptesh Kanojia, Prashant Sharma, Sayali Ghodekar, Pushpak Bhattacharyya, Gholamreza Haffari, and Malhar Kulkarni. 2021. Cognition-aware cognate detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3281–3292, On… find
  10. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. Searching for COMETINHO: The little metric that could. In Proceedings of the 23rd Annual Conference of the European Association for Machine Tran… find
  11. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702. Association for Computational Linguistics.
  12. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. A find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/willoughby-2026-surreycts-bea-2026-shared-task/