Elamur Research⌕
☰
home / papers / 2026 / Binary Token-Level Classification with DeBERTa for All-Type MWE Identification: A Lightweight Approach with Linguistic Enhancement

Binary Token-Level Classification with DeBERTa for All-Type MWE Identification: A Lightweight Approach with Linguistic Enhancement

Paper · 2026 · Diego Rossini and Lonneke van der Plas

Summary

The paper addresses the identification of multiword expressions (MWEs) of all types in English. Discontinuous cases such as look … up are especially hard, and the strongest earlier results came from very large LLMs. The authors fine-tune DeBERTa-v3-large to make three separate yes/no decisions for each token: whether it starts, ends, or lies inside an MWE. Spans are rebuilt from these predictions using thresholds tuned on development data. This design grows linearly with sentence length, unlike span enumeration.

The model also receives two syntactic signals as learned embeddings: NP-chunk membership and dependency-path distances. Dependency distances additionally filter out implausible discontinuous candidates. Class imbalance is handled through data augmentation.

On CoAM, the system reaches 69.8% F1, beating Qwen-72B by about 12 points with 165× fewer parameters. It also generalizes to STREUSLE, with 78.9% F1. The work shows that a compact, linguistically informed encoder can outperform LLMs on structured tasks of this kind, which is relevant when computing resources are limited.

Key points

  • Datasets: CoAM has 1,301 sentences (780 train / 521 test) and 867 MWEs with type labels (NOUN, VERB, MOD/CONN, CLAUSE, OTHER). STREUSLE v4.6 (UD v2.16) was remapped to CoAM-compatible types.
  • Linguistic features:
    • NP chunks from spaCy en_core_web_lg, encoded as 16-dim embeddings.
    • Dependency distances computed with NetworkX and capped at 5, encoded as 32-dim embeddings.
    • During reconstruction, discontinuous candidates are dropped if consecutive members are more than 4 dependency steps apart.
  • Augmentation depends on dataset size:
    • On CoAM, oversampling 30% of sentences containing MWEs worked best.
    • On the larger STREUSLE, lexical substitution worked better.
  • Discontinuous MWEs remain hard:
    • Recall roughly doubled compared with Qwen-72B (34.9% vs 17.1%).
    • F1 is still only 29.7%.
  • Release and limitations:
    • The model is available on Hugging Face (DiegoRossini/mwe-detection-deberta).
    • It predicts span boundaries but not MWE types.
    • It has been evaluated on English only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Università della Svizzera italianaLugano · Switzerland6 works · 647 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Idioms & multiword expressions · 217 works

Cite

Cite as Diego Rossini and Lonneke van der Plas. 2026. Binary Token-Level Classification with DeBERTa for All-Type MWE Identification: A Lightweight Approach with Linguistic Enhancement. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.135

@inproceedings{rossini2026binary,
  title = {{Binary Token-Level Classification with DeBERTa for All-Type MWE Identification: A Lightweight Approach with Linguistic Enhancement}},
  author = {Diego Rossini and Lonneke {van der Plas}},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.135},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.135}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Timothy Baldwin and Su Nam Kim. 2010. Multiword expressions. In Handbook of Natural Language Processing, pages 267–292. CRC Press. find
  2. Mathieu Constant, Gülşen Eryiğit, Johanna Monti, Lonneke van der Plas, Carlos Ramisch, Michael Rosner, and Amalia Todirascu. 2017. Multiword expression processing: A survey. Computational Linguistics, 43(4):837–892. find
  3. Silvio Cordeiro, Carlos Ramisch, and Aline Villavicencio. 2016. LIF&UFRGS at SemEval-2016 Task 10: Rule-based MWE identification and predominant-supersense tagging. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 910–917. Association for Computational L… find
  4. Markus Eberts and Adrian Ulges. 2020. Span-based joint entity and relation extraction with transformer pre-training. In 24th European Conference on Artificial Intelligence (ECAI 2020), pages 2006–2013. IOS Press. find
  5. Hao Fei, Yafeng Ren, and Donghong Ji. 2021. Boundary enhanced neural span classification for nested named entity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9016–9023. find
  6. Spence Green, Marie-Catherine de Marneffe, and Christopher D. Manning. 2013. Parsing models for identifying multiword expressions. Computational Linguistics, 39(1):195–227.
  7. Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. 2008. NetworkX: Network analysis in Python. In Proceedings of the 7th Python in Science Conference (SciPy 2008), pages 11–15. find
  8. Haibo He and Edwardo A. Garcia. 2009. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9):1263–1284. find
  9. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with disentangled attention. In Proceedings of the International Conference on Learning Representations (ICLR).
  10. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength natural language processing in Python. find
  11. Yusuke Ide, Joshua Tanner, Adam Nohejl, Jacob Hoffman, Justin Vasselli, Hidetaka Kamigaito, and Taro Watanabe. 2025. CoAM: Corpus of all-type multiword expressions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 27004– 27021,… find
  12. Yongliang Li, Xinyu Ma, Xiaoqing Zhou, and Jizhong Qi. 2023. T2-ner: A two-stage span-based framework for unified named entity recognition with templates. Transactions of the Association for Computational Linguistics, 11:224–240. find
  13. Dekang Lin. 1999. Automatic identification of non-compositional phrases. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics, pages 317–324.
  14. Alexis Nasr, Carlos Ramisch, José Deulofeu, and André Valli. 2015. Joint dependency parsing and multiword expression tokenization. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pag… find
  15. Carlos Ramisch, Silvio Ricardo Cordeiro, Agata Savary, Veronika Vincze, Verginica Barbu Mititelu, Archna Bhatia, Maja Buljan, Marie Candito, Polona Gantar, Voula Giouli, and 1 others. 2018. Edition 1.1 of the PARSEME shared task on automatic identification of verbal multiword expressions. In Proceed…
  16. Carlos Ramisch, Agata Savary, Bruno Guillaume, Jakub Waszczuk Abigail Walsh, Marie Candito, Ashim Gupta Mehmet Kurfalí, Verginica Barbu Mititelu, Gözde Berk, Johanna Monti, and 1 others. 2020. find
  17. Edition 1.2 of the PARSEME shared task on semi-supervised identification of verbal multiword expressions. In Proceedings of the Joint Workshop on Multiword Expressions and Electronic Lexicons, pages 107–118. Association for Computational Linguistics.
  18. Chuanqi Tan, Wei Qiu, Mosha Chen, Rui Wang, and Fei Huang. 2020. Boundary enhanced neural span classification for nested named entity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9016–9023. find
  19. Lance A. Ramshaw and Mitchell P. Marcus. 1995. Text chunking using transformation-based learning. In Third Workshop on Very Large Corpora, pages 82– 94. Association for Computational Linguistics.
  20. Joshua Tanner and Jacob Hoffman. 2023. MWE as WSD: Solving multiword expression identification with word sense disambiguation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 181–193. Association for Computational Linguistics.
  21. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147–155.
  22. Omid Rohanian, Shiva Taslimipoor, Samaneh Kouchaki, Le An Ha, and Ruslan Mitkov. 2019. Bridging the gap: Attending to discontinuity in identification of multiword expressions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  23. Kevin Walsh, Carlos Ramisch, Agata Savary, and Silvio Cordeiro. 2018. A dataset of verbal multiword expressions annotated with syntactic variants. In Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018), pages 101–110. Association for… find
  24. Eric Wehrli. 2000. Incremental parsing and multiword expression. In Proceedings of the Workshop on Multiword Expressions: Analysis, Acquisition and Treatment, pages 33–40. find
  25. Ivan A Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions: A pain in the neck for NLP. In International conference on intelligent text processing and computational linguistics, pages 1–15. Springer.
  26. Agata Savary, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova, and Antoine Doucet. 2017. The PARSEME shared task on automatic identification of verbal multiword expressions. In Proceedings of the 13…
  27. Nathan Schneider, Dirk Hovy, Anders Johannsen, and Marine Carpuat. 2016. SemEval-2016 Task 10: Detecting minimal semantic units and their meanings (DiMSUM). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 546–559, San Diego, California. Association for…
  28. Nathan Schneider, Jakob Prange, Emily Danchik, Luke Gessler, Samantha Onuffer, Nora Mordowanec, Nour Kazour, and Aviram Nguyen. 2018. Comprehensive supersense disambiguation of English prepositions and possessives. In Proceedings of the 56th Annual Meeting of the Association for Computational Lingui… find
  29. Raghavendra Swaminathan and Paul Cook. 2023. Token-level identification of multiword expressions using pre-trained multilingual language models. In Proceedings of the 19th Workshop on Multiword Expressions (MWE 2023), pages 1–6, Dubrovnik, Croatia. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rossini-2026-binary-token-level-classification-deberta/