Elamur Research⌕
☰
home / papers / 2016 / SemEval-2016 Task 10: Detecting Minimal Semantic Units and their Meanings (DiMSUM)

SemEval-2016 Task 10: Detecting Minimal Semantic Units and their Meanings (DiMSUM)

Summary

DiMSUM, a SemEval-2016 shared task, asks systems to do broad-coverage lexical semantic analysis of English. Systems must jointly find multiword expressions (MWEs), including discontinuous ones, and assign coarse WordNet supersenses to nouns and verbs. The goal is a middle ground between two existing options: fine-grained WordNet sense disambiguation, which is costly to annotate and generalizes poorly, and NER, which covers only some nouns.

Each token receives a paired label. The first part is a 6-position BIO-style MWE marker that can encode gaps. The second is one of 41 supersenses, and strong MWEs are labeled as single units.

Training data merged existing annotated online reviews and tweets. A newly annotated test set adds reviews, tweets and TED talk transcripts, a genre absent from training.

Nine systems took part, and the best composite F1 was only 57.7%. Error analysis found three things:

  • Many instances are solved by all systems or by none.
  • Scores barely differ across domains.
  • Systems largely make the same predictions.

The authors therefore conclude that new approaches are needed. The released data provides a multi-genre benchmark for coarse lexical semantics.

Key points

  • The tagset covers 26 noun and 15 verb supersenses (41 in total) and only strong MWEs. Tags follow a 6-tag scheme (O, o, B, b, I, i) in which lowercase marks tokens inside a gap.
  • Training data has 4,799 sentences and 73,826 words, drawn from STREUSLE 2.1 reviews plus Lowlands and Ritter tweets. The test set has 1,000 sentences and 16,500 words from Trustpilot, Tweebank, and TED (NAIST-NTT and IWSLT test). All data is POS-tagged with UD 1.2-style UPOS and published at github.com/dimsum16/dimsum-data.
  • One linguist annotated the test data in about 50 hours. A partial double annotation gave inter-annotator agreement of 60–75% F1 for MWEs and 67–80% accuracy for supersenses.
  • Out of 9 submitted systems, the top composite F1 averaged over three domains was 57.7%. TED was not clearly harder than the domains seen in training.
  • About 25% of gold supersense instances are missed by essentially every system, so even an oracle combining all systems would stay below 75% accuracy. Labels such as V.STATIVE are easy, while N.ATTRIBUTE and V.CONTACT are hard.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

45 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 35 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s16-1084

@inproceedings{schneider2016semeval,
  title = {{SemEval-2016 Task 10: Detecting Minimal Semantic Units and their Meanings (DiMSUM)}},
  author = {Schneider, Nathan and Hovy, Dirk and Johannsen, Anders and Carpuat, Marine},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1084},
  url = {https://doi.org/10.18653/v1/s16-1084}
}

References

50 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Giuseppe Attardi, Stefano Dei Rossi, Giulia Di Pietro, Alessandro Lenci, Simonetta Montemagni, and Maria Simi. 2010. A resource and tool for super-sense tagging of Italian texts. In Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner, and Da… find
  2. Timothy Baldwin and Su Nam Kim. 2010. Multiword expressions. In Nitin Indurkhya and Fred J. Damerau, editors, Handbook of Natural Language Processing, Second Edition. CRC Press, Taylor and Francis Group, Boca Raton, FL. find
  3. Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English Web Treebank. Technical Report LDC2012T13, Linguistic Data Consortium, Philadelphia, PA. URL http://www.ldc.upenn.edu/Catalog/catalogEntry. jsp?catalogId=LDC2012T13 . tronic lexical database. MIT Press, Cambridge, MA. ↗
  4. Spence Green, Marie-Catherine de Marneffe, and Christopher D. Manning. 2012. Parsing models for identifying multiword expressions. Computational Linguistics, 39(1):195–227. find
  5. Michael Heilman. 2011. Automatic factual question generation from text. Ph.D. dissertation, Carnegie Mellon University, Pittsburgh, Pennsylvania. URL http://www.ark.cs.cmu.edu/mheilman/questions/papers/ heilman-question-generation-dissertation.pdf . ↗
  6. Mohammad Javad Hosseini, Noah A. Smith, and Su-In Lee. 2016. UW-CSE at SemEval-2016 Task 10: Detecting multiword expressions and supersenses using double-chained conditional random fields. In Proc. of SemEval. San Diego, California, USA. find
  7. Jari Björne and Tapio Salakoski. 2016. UTU at SemEval-2016 Task 10: Binary Classification for Expression Detection (BCED). In Proc. of SemEval. San Diego, California, USA. find
  8. Dirk Hovy, Anders Johannsen, and Anders Søgaard. 2015. User review sites as a resource for large-scale sociolinguistic studies. In Proc. of WWW, pages 452– 461. Florence, Italy. find
  9. Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467–479. find
  10. Dirk Hovy, Shashank Shrivastava, Sujay Kumar Jauhar, Mrinmaya Sachan, Kartik Goyal, Huying Li, Whitney Sanders, and Eduard Hovy. 2013. Identifying metaphorical word use with tree kernels. In Proc. of the First Workshop on Metaphor in NLP, pages 52–57. Atlanta, Georgia, USA. find
  11. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT 3 : Web Inventory of Transcribed and Translated Talks. In Mauro Cettolo, Marcello Federico, Lucia Specia, and Andy Way, editors, Proc. of EAMT, pages 261–268. Trento, Italy. find
  12. Dirk Hovy and Anders Søgaard. 2015. Tagging performance correlates with author age. In Proc. of ACL-IJCNLP, pages 483–488. Beijing, China. find
  13. Massimiliano Ciaramita and Yasemin Altun. 2006. Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger. In Proc. of EMNLP, pages 594–602. Sydney, Australia. find
  14. Dirk Hovy, Stephen Tratz, and Eduard Hovy. 2010. What’s in a preposition? Dimensions of sense disambiguation for an interesting word class. In Coling 2010: Posters, pages 454–462. Beijing, China. find
  15. Matthieu Constant and Anthony Sigogne. 2011. MWU-aware part-of-speech tagging with a CRF model and lexical resources. In Proc. of the Workshop on Multiword Expressions: from Parsing and Generation to the Real World, pages 49–56. Portland, Oregon, USA. find
  16. Anders Johannsen, Dirk Hovy, Héctor Martínez Alonso, Barbara Plank, and Anders Søgaard. 2014. More or less supervised supersense tagging of Twitter. In Proc. of *SEM, pages 1–11. Dublin, Ireland. find
  17. Silvio Cordeiro, Carlos Ramisch, and Aline Villavicencio. 2016. UFRGS&LIF at SemEval-2016 Task 10: Rule-based MWE identification and predominant-supersense tagging. In Proc. of SemEval. San Diego, California, USA. find
  18. Christiane Fellbaum, editor. 1998. WordNet: an elec-Angelika Kirilin, Felix Krauss, and Yannick Versley. 2016. ICL-HD at SemEval-2016 Task 10: Improving the detection of minimal semantic units and their meanings with an ontology and word embeddings. In Proc. of SemEval. San Diego, California, USA. find
  19. Lingpeng Kong, Nathan Schneider, Swabha Swayamdipta, Archna Bhatia, Chris Dyer, and Noah A. Smith. 2014. A dependency parser for tweets. In Proc. of EMNLP, pages 1001–1012. Doha, Qatar. find our card
  20. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
  21. Héctor Martínez Alonso, Anders Johannsen, Sussi Olsen, Sanni Nimb, Nicolai Hartvig Sørensen, Anna Braasch, Anders Søgaard, and Bolette Sandford Pedersen. 2015. Supersense tagging for Danish. In Beáta Megyesi, editor, Proc. of NODALIDA, pages 21–29. Vilnius, Lithuania. find
  22. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Proc. of HLT, pages 303–308. Plainsboro, NJ, USA. find
  23. Graham Neubig, Katsuhito Sudoh, Yusuke Oda, Kevin Duh, Hajime Tsukada, and Masaaki Nagata. 2014. The naist-ntt ted talk treebank. In International Workshop on Spoken Language Translation. find
  24. Joakim Nivre, Željko Agić, Maria Jesus Aranzabe, Masayuki Asahara, Aitziber Atutxa, Miguel Ballesteros, John Bauer, Kepa Bengoetxea, Riyaz Ahmad Bhat, Cristina Bosco, Sam Bowman, Giuseppe G. A. Celano, Miriam Connor, Marie-Catherine de Marneffe, Arantza Diaz de Ilarraza, Kaja Dobrovoljc, Timothy Do… find
  25. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In Proc. of NAACL-HLT, pages 380–390. Atlanta, Georgia, USA. find our card
  26. Gerhard Paaß and Frank Reichartz. 2009. Exploiting semantic constraints for estimating supersenses with CRFs. In Proc. of the Ninth SIAM International Conference on Data Mining, pages 485–496. Sparks, Nevada. find
  27. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. arXiv:1104.2086 [cs]. URL http://arxiv.org/abs/1104.2086 . ↗
  28. Davide Picca, Alfio Massimiliano Gliozzo, and Simone Campora. 2009. Bridging languages by SuperSense entity tagging. In Proc. of NEWS, pages 136–142. Suntec, Singapore. find
  29. Davide Picca, Alfio Massimiliano Gliozzo, and Massimiliano Ciaramita. 2008. Supersense Tagger for Italian. In Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, and Daniel Tapias, editors, Proc. of LREC, pages 2386–2390. Marrakech, Morocco. find
  30. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Learning part-of-speech taggers with inter-annotator agreement loss. In Proc. of EACL, pages 742–751. Gothenburg, Sweden. find
  31. Likun Qiu, Yunfang Wu, Yanqiu Shao, and Alexander Gelbukh. 2011. Combining contextual and structural information for supersense tagging of Chinese unknown words. In Computational Linguistics and Intelligent Text Processing: Proceedings of the 12th International Conference (CICLing’11), volume 6608 o… find
  32. Carlos Ramisch, Vitor De Araujo, and Aline Villavicencio. 2012. A broad evaluation of techniques for automatic acquisition of multiword expressions. In Proc. of ACL 2012 Student Research Workshop, pages 1–6. Jeju Island, Korea. find
  33. Lance A. Ramshaw and Mitchell P. Marcus. 1995. Text chunking using transformation-based learning. In Proc. of the Third ACL Workshop on Very Large Corpora, pages 82–94. Cambridge, MA. find
  34. Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: an experimental study. In Proc. of EMNLP, pages 1524–1534. Edinburgh, Scotland, UK. find
  35. Stefano Dei Rossi, Giulia Di Pietro, and Maria Simi. 2013. Description and results of the SuperSense tagging task. In Bernardo Magnini, Francesco Cutugno, Mauro Falcone, and Emanuele Pianta, editors, Evaluation of Natural Language and Speech Tools for Italian, number 7689 in Lecture Notes in Compute… find
  36. Martha Palmer. 2015. A hierarchy with, of, and for preposition supersenses. In Proc. of The 9th Linguistic Annotation Workshop, pages 112–123. Denver, Colorado, USA. find our card
  37. Xin Tang, Fei Li, and Donghong Ji. 2016. WHUNlp at SemEval-2016 Task 10: A pilot study in detecting minimal semantic units and their meanings using supervised models. In Proc. of SemEval. San Diego, California, USA. find
  38. Stephen Tratz and Eduard Hovy. 2010. ISI: Automatic classification of relations between nominals using a maximum entropy classifier. In Proc. of SemEval, pages 222–225. Uppsala, Sweden. find
  39. Andreas Scherbakov, Ekaterina Vylomova, Fei Liu, and Timothy Baldwin. 2016. VectorWeavers at SemEval-2016 Task 10: From incremental meaning to semantic unit (phrase by phrase). In Proc. of SemEval. San Diego, California, USA. find
  40. Yulia Tsvetkov, Elena Mukomel, and Anatole Gershman. 2013. Cross-lingual metaphor detection using common semantic features. In Proc. of the First Workshop on Metaphor in NLP, pages 45–51. Atlanta, Georgia, USA. find
  41. Nathan Schneider, Emily Danchik, Chris Dyer, and Noah A. Smith. 2014a. Discriminative lexical semantic segmentation with gaps: running the MWE gamut. Transactions of the Association for Computational Linguistics, 2:193–206. find our card
  42. Yulia Tsvetkov, Nathan Schneider, Dirk Hovy, Archna Bhatia, Manaal Faruqui, and Chris Dyer. 2014. Augmenting English adjective senses with supersenses. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Pi… find
  43. Nathan Schneider, Behrang Mohit, Chris Dyer, Kemal Oflazer, and Noah A. Smith. 2013. Supersense tagging for Arabic: the MT-in-the-middle attack. In Proc. of NAACL-HLT, pages 661–667. Atlanta, Georgia, USA. find
  44. Nathan Schneider, Behrang Mohit, Kemal Oflazer, and Noah A. Smith. 2012. Coarse lexical semantic annotation with supersenses: an Arabic case study. In Proc. of ACL, pages 253–258. Jeju Island, Korea. find our card
  45. Nathan Schneider, Spencer Onuffer, Nora Kazour, Emily Danchik, Michael T. Mordowanec, Henrietta Conrad, and Noah A. Smith. 2014b. Comprehensive annotation of multiword expressions in a social web corpus. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph… find our card
  46. Nathan Schneider and Noah A. Smith. 2015. A corpus and model integrating multiword expressions and supersenses. In Proc. of NAACL-HLT, pages 1537–1547. Denver, Colorado. find our card
  47. Nathan Schneider, Vivek Srikumar, Jena D. Hwang, and Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proc. of MUC-6, pages 45–52. Columbia, Maryland, USA. find our card
  48. Veronika Vincze, István Nagy T., and Gábor Berend. 2011. Detecting noun compounds and light verb constructions: a contrastive study. In Proc. of the Workshop on Multiword Expressions: from Parsing and Generation to the Real World, pages 116–121. Portland, Oregon, USA. find
  49. Veronika Vincze, János Zsibrita, and István Nagy T. 2013. Dependency parsing for identifying Hungarian light verb constructions. In Proc. of the Sixth International Joint Conference on Natural Language Processing, pages 207–215. Nagoya, Japan. find
  50. Patrick Ye and Timothy Baldwin. 2007. MELB-YB: Preposition sense disambiguation using rich semantic features. In Proc. of SemEval, pages 241–244. Prague, Czech Republic. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/schneider-2016-semeval-2016-task-10-detecting/