Elamur Research⌕
☰
home / papers / 2016 / SemEval-2016 Task 14: Semantic Taxonomy Enrichment

SemEval-2016 Task 14: Semantic Taxonomy Enrichment

Paper · 2016 · David Jurgens and Mohammad Taher Pilehvar

Summary

WordNet is widely used in NLP, but it is curated by hand, updates slowly, and misses many domain terms, slang and neologisms. Earlier enrichment methods were mostly judged on word-level relations, not on whether a new sense lands at the correct concept. SemEval-2016 Task 14 addresses this with a shared evaluation. Systems receive a new lemma, its part of speech and a definition, drawn mostly from Wiktionary and other glossaries. They must choose a WordNet 3.0 synset and one of two operations: MERGE (add the item as a synonym of that synset) or ATTACH (create a new hyponym under it). The organizers built 1,000 annotated noun and verb items covering genres such as medical, legal, slang and finance. Five teams entered 13 systems, and every one beat a random baseline. Only one system narrowly beat a simple heuristic that finds the earliest word in the gloss whose part of speech matches and links the new item under that word's sense number one. This shows that exploiting how definitions are structured matters, and that taxonomy enrichment still has much room to improve.

Key points

  • The dataset has 1,000 items: 400 for training and 600 for testing (516 nouns and 84 verbs in test). Most items require ATTACH; only 32 test items are MERGE.
  • There were two participation tracks. Resource-aware systems could use the source dictionaries, such as Wiktionary pages and their markup. Constrained systems could use any resource except dictionaries.
  • The two authors initially agreed exactly on only 37.5% of items. Their disagreements were still semantically close, with an average Wu&P similarity of 0.74, and all were adjudicated.
  • The best entry, System2 of MSejrKU, reached F1 0.680, essentially tied with the FWFS baseline at about 0.679. The random baseline scored about 0.370. Lemma Match and Wu&P scores were highly correlated (r=0.96).
  • Systems struggled most on the Finance and Jargon genres, which had little training data. They still did well on Legal, a surprise genre with no training data. The authors plan to feed the task's insights into CROWN, an automatically extended WordNet.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States567 works · 335,189 citations🎓University of CambridgeCambridge · United Kingdom287 works · 125,174 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as David Jurgens and Mohammad Taher Pilehvar. 2016. SemEval-2016 Task 14: Semantic Taxonomy Enrichment. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s16-1169

@inproceedings{jurgens2016semeval,
  title = {{SemEval-2016 Task 14: Semantic Taxonomy Enrichment}},
  author = {David Jurgens and Mohammad Taher Pilehvar},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1169},
  url = {https://doi.org/10.18653/v1/s16-1169}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre and Aitor Soroa. 2009. Personalizing pagerank for word sense disambiguation. In Proceed-https://github.com/davidjurgens/crown ings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, pages 33–41. Association for Computational Linguistics. find
  2. Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proce…
  3. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. sem 2013 shared task: Semantic textual similarity, including a pilot on typed-similarity. In In* SEM 2013: The Second Joint Conference on Lexical and Computational Semantics. Association for Computational Linguistics.…
  4. Georgeta Bordea, Paul Buitelaar, Stefano Faralli, and Roberto Navigli. 2015. SemEval-2015 task 17: Taxonomy extraction evaluation (texeval). In Proceedings of the 9th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  5. Georgeta Bordea, Els Lefever, and Paul Buitelaar. 2016. SemEval-2016 task 13: Taxonomy extraction evaluation (texeval-2). In Proceedings of the 10th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find
  6. Alexander Budanitsky and Graeme Hirst. 2006a. Evaluating WordNet-based measures of Lexical Semantic Relatedness. Computational Linguistics, 32(1):13–47.
  7. Alexander Budanitsky and Graeme Hirst. 2006b. Evaluating WordNet-based measures of lexical semantic relatedness. Computational Linguistics, 32(1):13–47.
  8. Andrea Esuli and Fabrizio Sebastiani. 2006. SENTI-WORDNET: A publicly available lexical resource for opinion mining. In In Proceedings of the 5th Conference on Language Resources and Evaluation, pages 417–422. find
  9. Hui Fang. 2008. A re-examination of query expansion using lexical resources. In Proceedings of ACL-08: HLT, pages 139–147, Columbus, Ohio. Christiane Fellbaum, editor. 1998. find
  10. WordNet: An Electronic Database. MIT Press, Cambridge, MA. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: the 90% solution. In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, pages 57–60. Associ… find
  11. Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2015. SensEmbed: learning sense embeddings for word and relational similarity. In Proceedings of ACL, pages 95–105.
  12. David Jurgens and Ioannis Klapaftis. 2013. SemEval-2013 task 13: Word sense induction for graded and non-graded senses. In Second joint conference on lexical and computational semantics (* SEM), volume 2, pages 290–299. find
  13. David Jurgens and Mohammad Taher Pilehvar. 2015. Reserating the awesometastic: An automatic extension of the WordNet taxonomy for novel terms. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1459… find
  14. David Jurgens, Mohammad Taher Pilehvar, and Roberto Navigli. 2014. SemEval-2014 task 3: Cross-level semantic similarity. In Proceedings of the 8 th International Workshop on Semantic Evaluation (SemEval 2014), in conjunction with COLING 2014, pages 17– 26, Dublin, Ireland. find
  15. Jey Han Lau, Paul Cook, Diana McCarthy, David Newman, and Timothy Baldwin. 2012. Word sense induction for novel sense detection. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 591–601. Association for Computational Linguistics.
  16. Suresh Manandhar, Ioannis P Klapaftis, Dmitriy Dligach, and Sameer S Pradhan. 2010. SemEval-2010 task 14: Word sense induction & disambiguation. In Proceedings of the 5th international workshop on semantic evaluation, pages 63–68. Association for Computational Linguistics.
  17. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The stanford corenlp natural language processing toolkit. In ACL (System Demonstrations), pages 55–60.
  18. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217–250. find
  19. Roberto Navigli and Daniele Vannella. 2013. SemEval-2013 task 11: Word sense induction and disambiguation within an end-user application. In Second Joint Conference on Lexical and Computational Semantics (* SEM), volume 2, pages 193–201. find
  20. Roberto Navigli. 2009. Word Sense Disambiguation: A survey. ACM Computing Surveys, 41(2):1–69. find
  21. Mohammad Taher Pilehvar, David Jurgens, and Roberto Navigli. 2013. Align, Disambiguate and Walk: a Unified Approach for Measuring Semantic Similarity. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL), pages 1341–1351, Sofia, Bulgaria. find
  22. Michael Poprat, Elena Beisswanger, and Udo Hahn. 2008. Building a BioWordNet by using WordNet’s data formats and WordNet’s software infrastructure: a failure story. In Proceedings of the Workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pages 31–39, Co… find
  23. Rion Snow, Dan Jurafsky, and Andrew Ng. 2006. Semantic taxonomy induction from heterogeneous evidence. pages 801–808. find
  24. Antonio Toral, Rafael Muoz, and Monica Monachini. 2008. Named Entity WordNet. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC), pages 741–747. find
  25. Yulia Tsvetkov, Nathan Schneider, Dirk Hovy, Archna Bhatia, Manaal Faruqui, and Chris Dyer. 2014. Augmenting English adjective senses with supersenses. In LREC.
  26. European Language Resources Association. Giannis Varelas, Epimenidis Voutsakis, Paraskevi Raftopoulou, Euripides G.M. Petrakis, and Evangelos E. Milios. 2005. Semantic similarity methods in WordNet and their application to information retrieval on the web. In Proceedings of the 7th Annual ACM Intern… find
  27. Zhibiao Wu and Martha Palmer. 1994. Verbs semantics and lexical selection. In Proceedings of the 32Nd Annual Meeting on Association for Computational Linguistics, pages 133–138. find
  28. Ichiro Yamada, Jong-Hoon Oh, Chikara Hashimoto, Kentaro Torisawa, Junichi Kazama, Stijn De Saeger, and Takuya Kawada. 2011. Extending WordNet with hypernyms and siblings acquired from wikipedia. In Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 874–882. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jurgens-2016-semeval-2016-task-14-semantic/