Elamur Research⌕
☰
home / papers / 2020 / Edition 1.2 of the PARSEME Shared Task on Semi-supervised Identification of Verbal Multiword Expressions

Edition 1.2 of the PARSEME Shared Task on Semi-supervised Identification of Verbal Multiword Expressions

Summary

Verbal multiword expressions (VMWEs), such as idioms, light verb constructions and particle verbs, must be located in text before deeper semantic analysis. Earlier PARSEME campaigns found that ambiguity between idiomatic and literal readings is rarely the real obstacle. The hard part is detecting expressions that never appeared in the annotated training material. Edition 1.2 therefore reorganises the shared task around these unseen VMWEs.

The organisers released VMWE-annotated corpora for 14 languages, including new data for Chinese and Irish and a freshly annotated Swedish corpus. All morphosyntactic layers were aligned with UD version 2. Each test set was built to contain about 300 expressions absent from both train and dev, and raw corpora were supplied to encourage semi-supervised discovery. Seven teams submitted nine runs.

The strongest open-track system clearly exceeded the best unseen score from edition 1.1, though the organisers caution that the two editions are not strictly comparable. The analysis also shows that a language's proportion of unseen expressions predicts system quality far better than training set size, which is useful guidance for building future resources.

Key points

  • Coverage and participation: The data cover 14 languages (EU, DE, EL, GA, FR, HI, HE, PL, IT, RO, PT, TR, SV, ZH) and can be downloaded from hdl.handle.net/11234/1-3367. Seven teams submitted 9 results: 2 in the closed track and 7 in the open track.
  • Definition of "unseen": A VMWE counts as unseen if its multiset of lexicalised lemmas was never annotated in train or dev combined. Each test set holds roughly 300 such items. Morphosyntax is either manual or produced by UDPipe trained on UD 2.5.
  • Unseen-VMWE results: MTLB-STRUCT reached an unseen F1 of 38.53 in the open track, compared with 28.46 for SHOMA in edition 1.1. TRAVIS-mono achieved 60.19 on Chinese, where a simple two-rule baseline already reaches 27.33.
  • Global results: Overall MWE-based F1 exceeded 66 in the closed track and 70 in the open track, compared with 54 and 58 in edition 1.1. Seen2Seen, a rule-based closed-track system, ranked second overall across both tracks.
  • What predicts performance: The best F1 per language correlates strongly with that language's unseen ratio (Pearson −0.90). It correlates only weakly with training size (0.23).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

42 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 38 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ramisch2020edition,
  title = {{Edition 1.2 of the PARSEME Shared Task on Semi-supervised Identification of Verbal Multiword Expressions}},
  author = {Ramisch, Carlos and Savary, Agata and Guillaume, Bruno and Waszczuk, Jakub and Candito, Marie and Vaidya, Ashwini and Mititelu, Verginica Barbu and Bhatia, Archna and Iñurrieta, Uxoa and Giouli, Voula and Güngör, Tunga and Jiang, Menghan and Lichte, Timm and Liebeskind, Chaya and Monti, Johanna and Ramisch, Renata and Stymne, Sara and Walsh, Abigail and Xu, Hongzhi},
  booktitle = {Proceedings of the Joint Workshop on Multiword Expressions and Electronic Lexicons},
  year = {2020},
  doi = {10.48448/1wgq-xb32},
  url = {https://doi.org/10.48448/1wgq-xb32}
}

References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hazem Al Saied, Marie Candito, and Mathieu Constant. 2018. A transition-based verbal multiword expression analyzer. In Stella Markantonatou, Carlos Ramisch, Agata Savary, and Veronika Vincze, editors, Multiword expressions at length and in depth. Language Science Press. find
  2. Timothy Baldwin and Su Nam Kim. 2010. Multiword expressions. In Nitin Indurkhya and Fred J. Damerau, editors, Handbook of Natural Language Processing, Second Edition, pages 267–292. CRC Press, Taylor and Francis Group, Boca Raton, FL. ISBN 978-1420085921. find
  3. Nicoletta Calzolari, Charles Fillmore, Ralph Grishman, Nancy Ide, Alessandro Lenci, Catherine MacLeod, and Antonio Zampolli. 2002. Towards best practice for multiword expressions in computational lexicons. In Proceedings of the 3rd International Conference on Language Resources and Evaluation (LREC-… find
  4. Mathieu Constant, Gülşen Eryiğit, Johanna Monti, Lonneke van der Plas, Carlos Ramisch, Michael Rosner, and Amalia Todirascu. 2017. Multiword expression processing: A survey. Computational Linguistics, 43(4):837– 892. find
  5. Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140–147, Copenhagen, Denmark, September. Association for Computational Lingu… find
  6. Carlos Ramisch, Silvio Ricardo Cordeiro, Agata Savary, Veronika Vincze, Verginica Barbu Mititelu, Archna Bhatia, Maja Buljan, Marie Candito, Polona Gantar, Voula Giouli, Tunga Güngör, Abdelati Hawwari, Uxoa Iñurrieta, Jolanta Kovalevskaitė, Simon Krek, Timm Lichte, Chaya Liebeskind, Johanna Mont… find our card
  7. Agata Savary, Carlos Ramisch, Silvio Cordeiro, Federico Sangati, Veronika Vincze, Behrang QasemiZadeh, Marie Candito, Fabienne Cap, Voula Giouli, Ivelina Stoyanova, and Antoine Doucet. 2017. The PARSEME shared task on automatic identification of verbal multiword expressions. In Proceedings of the 13… find our card
  8. Agata Savary, Marie Candito, Verginica Barbu Mititelu, Eduard Bejček, Fabienne Cap, Slavomı́r Čéplö, Silvio Ricardo Cordeiro, Gülşen Eryiğit, Voula Giouli, Maarten van Gompel, Yaakov HaCohen-Kerner, Jolanta Kovalevskaitė, Simon Krek, Chaya Liebeskind, Johanna Monti, Carla Parra Escartı́n, Lo… find
  9. Agata Savary, Silvio Cordeiro, and Carlos Ramisch. 2019a. Without lexicons, multiword expression identification will never fly: A position statement. In MWE-WN 2019, pages 79–91, Florence, Italy. ACL. find
  10. Agata Savary, Silvio Ricardo Cordeiro, Timm Lichte, Carlos Ramisch, Uxoa Iñurrieta, and Voula Giouli. 2019b. Literal occurrences of multiword expressions: Rare birds that cause a stir. The Prague Bulletin of Mathematical Linguistics, 112:5–54, apr. find
  11. Nathan Schneider, Dirk Hovy, Anders Johannsen, and Marine Carpuat. 2016. SemEval-2016 Task 10: Detecting Minimal Semantic Units and their Meanings (DiMSUM). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 546–559, San Diego, California, June. Associatio… find our card
  12. Milan Straka and Jana Straková. 2017. Tokenizing, pos tagging, lemmatizing and parsing ud 2.0 with udpipe. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 88–99, Vancouver, Canada, August. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/ramisch-2020-edition-1-2-parseme-shared/