Elamur Research⌕
☰
home / papers / 2023 / Token-level Identification of Multiword Expressions using Pre-trained Multilingual Language Models

Token-level Identification of Multiword Expressions using Pre-trained Multilingual Language Models

Summary

The paper asks whether pre-trained multilingual language models learn knowledge about multiword expressions (MWEs) and idiomaticity that carries over to languages absent from training. The authors revisit two shared tasks under new cross-lingual conditions. For SemEval 2022 task 2 subtask A, which classifies sentence-level usage as idiomatic or literal in the zero-shot setting, they fine-tune mBERT, XLM-RoBERTa and mDeBERTa on English only, Portuguese only, or both, then test on English, Portuguese and Galician. For PARSEME 1.2, which tags verbal MWEs at the token level in fourteen languages, they run MTLB-STRUCT. This system learns MWE tags jointly with dependency trees on top of mBERT. They test it in two setups: an "all" setting that pools every language's training data, and a "heldout" setting that trains on all languages except the target one. On unseen languages, models consistently outperform a most-frequent-class baseline. Pooling multilingual data also beats monolingual training. These results point to partly language-independent MWE knowledge, which matters for languages that lack annotated MWE resources.

Key points

  • SemEval data and transfer results: the training set has 3327 English and 1164 Portuguese instances. Test sets contain 916, 713 and 713 instances for en, pt and gl, scored with macro F1.
    • Training on English and testing on Portuguese beat in-language Portuguese training for mBERT (0.583 vs 0.578) and XLM-RoBERTa (0.590 vs 0.553), likely because the English training set is larger.
  • Galician: English-only models fell below the baseline (0.434). Portuguese-trained models did slightly better, possibly due to the Romance-language kinship. Training on en+pt worked best overall, peaking with mBERT at 0.550.
  • PARSEME 1.2 averages over the 14 languages: MWE-based F1 was 0.699 (Mono), 0.722 (All) and 0.331 (Heldout), against a baseline of 0.002. Token-based F1 was 0.738, 0.746 and 0.381. Pooled multilingual training gave the top average on every metric.
  • Heldout vs. monolingual: heldout scores were below monolingual ones for every language and metric. They still exceeded the baseline for each language.
  • Per-category heldout results: the best categories were LVC.full (~0.35), IRV (~0.31) and VID (~0.23), which are the most frequent VMWE types in the data. Categories such as IAV and MVC scored 0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.mwe-1.1

@inproceedings{swaminathan2023token,
  title = {{Token-level Identification of Multiword Expressions using Pre-trained Multilingual Language Models}},
  author = {Swaminathan, Raghuraman and Cook, Paul},
  booktitle = {Proceedings of the 19th Workshop on Multiword Expressions (MWE 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.mwe-1.1},
  url = {https://doi.org/10.18653/v1/2023.mwe-1.1}
}

References

19 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Timothy Baldwin and Su Nam Kim. 2010. Multiword expressions. In Nitin Indurkhya and Fred J. Damerau, editors, Handbook of Natural Language Processing, 2nd edition. CRC Press, Boca Raton, USA. find
  2. Milton King and Paul Cook. 2018. Leveraging distributed representations and lexico-syntactic fixedness for token-level prediction of the idiomaticity of English verb-noun combinations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)… find
  3. Gábor Berend. 2011. Opinion expression mining by exploiting keyphrase extraction. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 1162–1170, Chiang Mai, Thailand. Asian Federation of Natural Language Processing. find
  4. Marine Carpuat and Mona Diab. 2010. Task-based evaluation of multiword expressions: a pilot study in statistical machine translation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 242–245, Los Angeles,… find
  5. Murathan Kurfalı and Robert Östling. 2020. Disambiguation of potentially idiomatic expressions with contextual embeddings. In Proceedings of the Joint Workshop on Multiword Expressions and Electronic Lexicons, pages 85–94, online. Association for Computational Linguistics. find
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. find
  7. Changsheng Liu and Rebecca Hwa. 2018. Heuristically informed unsupervised idiom usage recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1723–1731, Brussels, Belgium. Association for Computational Linguistics. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  9. Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lingu… find our card
  10. Samin Fakharian and Paul Cook. 2021. Contextualized embeddings encode monolingual and cross-lingual knowledge of idiomaticity. In Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021), pages 23–32, Online. Association for Computational Linguistics. find
  11. Carlos Ramisch, Silvio Cordeiro, Agata Savary, Veronika Vincze, Verginica Barbu Mititelu, Archna Bhatia, Maja Buljan, Marie Candito, Polona Gantar, Voula Giouli, et al. 2018. Edition 1.1 of the parseme shared task on automatic identification of verbal multiword expressions. In Proceedings of the Joi… find our card
  12. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language p… find our card
  13. Carlos Ramisch, Agata Savary, Bruno Guillaume, Jakub Waszczuk, Marie Candito, Ashwini Vaidya, Verginica Barbu Mititelu, Archna Bhatia, Uxoa Iñurrieta, Voula Giouli, Tunga Güngör, Menghan Jiang, Timm Lichte, Chaya Liebeskind, Johanna Monti, Renata Ramisch, Sara Stymne, Abigail Walsh, and Hongzhi Xu.… find our card
  14. Alexander Rush. 2020. Torch-struct: Deep structured prediction library. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 335–342, Online. Association for Computational Linguistics. find
  15. Giancarlo Salton, Robert Ross, and John Kelleher. 2016. Idiom token classification using sentential distributed semantics. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 194–204, Berlin, Germany. Association for Computational… find our card
  16. Nathan Schneider and Noah A. Smith. 2015. A corpus and model integrating multiword expressions and supersenses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1537–1547, Denver, Colorado. Associ… find our card
  17. Shiva Taslimipoor, Sara Bahaadini, and Ekaterina Kochmar. 2020. MTLB-STRUCT @parseme 2020: Capturing unseen multiword expressions using multi-task learning and pre-trained masked language models. In Proceedings of the Joint Workshop on Multiword Expressions and Electronic Lexicons, pages 142–148, on… find
  18. Shiva Taslimipoor, Omid Rohanian, and Le An Ha. 2019. Cross-lingual transfer learning and multitask learning for capturing multiword expressions. In Proceedings of the Joint Workshop on Multiword Expressions and WordNet (MWE-WN 2019), pages 155– 161, Florence, Italy. Association for Computational Li… find
  19. Harish Tayyar Madabushi, Edward Gow-Smith, Marcos Garcia, Carolina Scarton, Marco Idiart, and Aline Villavicencio. 2022. SemEval-2022 task 2: Multilingual idiomaticity detection and sentence embedding. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 107… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/swaminathan-2023-token-level-identification-multiword-expressions/