Elamur Research⌕
☰
home / papers / 2026 / Token Titans at BEA 2026 Shared Task 1: Multilingual Lexical Complexity Prediction via Fine-Tuned XLM-RoBERTa with Ensemble Decoding

Token Titans at BEA 2026 Shared Task 1: Multilingual Lexical Complexity Prediction via Fine-Tuned XLM-RoBERTa with Ensemble Decoding

Paper · 2026 · Anubhab Parashar and Sandeep Mathias

Summary

This system paper presents the authors' entry in the BEA 2026 shared task that targets vocabulary difficulty prediction. The task asks systems to estimate how hard words are for English learners whose L1 is Spanish, German or Chinese. Each instance pairs an L1 word and its sentence context with an English clue and an English target word. The authors fine-tune a separate XLM-RoBERTa Large regressor for each language, using a linear head on [CLS]. All four fields are merged into one string joined by </s> separators. GLMM labels are z-score normalized per language. Two runs per language vary the learning rate, scheduler and seed, and German alone gets a weighted ensemble. The headline figures, RMSE 1.170 and Pearson 0.812, come from Spanish only, where the system beats the xlm-roberta-base baseline. For German and Chinese, the baseline is better on RMSE. The authors blame unstable fine-tuning of a large model on small data, with a small effective batch, low learning rates and no dropout on the head. The paper shows that a single hyperparameter setup does not transfer across typologically different languages.

Key points

  • Input format: L1_word </s> L1_context </s> en_clue </s> en_target. For Chinese, whitespace is collapsed before tokenization. Predictions are denormalized and clipped to [0.05, 5.0].
  • Training setup:
    • AdamW, fp16, effective batch size 8, 5 epochs, 500 warmup steps.
    • Learning rates of 1e-5 to 1.2e-5.
    • Linear or cosine schedulers, seeds 42 and 1337.
    • No external data or lexical resources.
  • Spanish results:
    • Run 1 reached RMSE 1.170 and r 0.812, against a baseline of 1.257 and 0.765.
    • Run 2 collapsed to r 0.043.
  • German results: the 0.6/0.4 ensemble only matched Run 2, at RMSE 1.569 and r 0.772. The baseline scored 1.258 and 0.773.
  • Chinese results:
    • The baseline's RMSE of 1.140 beat both runs, which scored 1.382 and 1.436.
    • Run 2 had the best Pearson, 0.777 against the baseline's 0.753.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Presidency University.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Anubhab Parashar and Sandeep Mathias. 2026. Token Titans at BEA 2026 Shared Task 1: Multilingual Lexical Complexity Prediction via Fine-Tuned XLM-RoBERTa with Ensemble Decoding. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.80

@inproceedings{parashar2026token,
  title = {{Token Titans at BEA 2026 Shared Task 1: Multilingual Lexical Complexity Prediction via Fine-Tuned XLM-RoBERTa with Ensemble Decoding}},
  author = {Anubhab Parashar and Sandeep Mathias},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.80},
  url = {https://doi.org/10.18653/v1/2026.bea-1.80}
}

References

13 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dennis Aumiller and Michael Gertz. 2022. UniHD at TSAR-2022 shared task: Is compute all we need for lexical simplification? In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), pages 251–258, Abu Dhabi, United Arab Emirates (Virtual). Association for Com… find
  2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  3. Edgar Dale and Jeanne S. Chall. 1948. A formula for predicting readability. Educational Research Bulletin, 27(1):11–28. find
  4. Benjamin Dutilleul, Mathis Debaillon, and Sandeep Mathias. 2024. ISEP_Presidency_University at MLSP 2024 shared task: Using GPT-3.5 to generate substitutes for lexical simplification. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages… find
  5. Matthew Shardlow, Fernando Alva-Manchego, Riza Batista-Navarro, Stefan Bott, Saul Calderon Ramirez, Rémi Cardon, Thomas François, Akio Hayakawa, Andrea Horbach, Anna Hülsing, Yusuke Ide, Joseph Marvin Imperial, Adam Nohejl, Kai North, Laura Occhipinti, Nelson Peréz Rojas, Nishat Raihan, Tharindu Ran… find
  6. The BEA 2024 shared task on the multilingual lexical simplification pipeline. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 571–589, Mexico City, Mexico. Association for Computational Linguistics. find
  7. Matthew Shardlow, Richard Evans, Gustavo Paetzold, and Marcos Zampieri. 2021. SemEval-2021 task 1: Lexical complexity prediction. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1–16. Association for Computational Linguistics.
  8. Matthew Shardlow, Marcos Zampieri, and Richard Evans. 2022. Predicting the complexity of words in context with lexical resources and transformer models. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), pages 91–100. Association for Computational Ling… find
  9. George-Eduard Zaharia, Radu Ion, and Dan Tufis. 2022. Exploring transformer models for lexical complexity prediction. In Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), pages 11–20. Association for Computational Linguistics. find
  10. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NeurIPS), pages 6402–6413. find
  11. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487–4496. Association for Computational Linguistics.
  12. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021. On the stability of fine-tuning BERT: Misconceptions, explanations, and strong baselines. In Proceedings of the 9th International Conference on Learning Representations (ICLR). find
  13. Sian North, Fernando Alva-Manchego, and Matthew Shardlow. 2022. Predicting multilingual sentence readability with transformer models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3054–3065. International Committee on Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/parashar-2026-token-titans-bea-2026-shared/