Elamur Research
☰
home / papers / 2010 / SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals

SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/S10-1006.pdf

Summary

This paper presents a shared evaluation for identifying which semantic relation links two marked nouns in a sentence. For example, in "tea from dried ginseng", the relation is Entity-Origin. It extends SemEval-1 Task 4, which used one binary dataset per relation. Here, systems must choose among nine relations plus OTHER and also determine argument direction, without being told which noun fills which slot. The organizers built the data in three rounds. They harvested candidate sentences from the Web with many search patterns. Two people then annotated each candidate independently, and disagreements were adjudicated or the example was dropped. The result is a corpus of 10,717 examples. Ten teams submitted 28 runs, scored by macro-averaged F1 with directionality. The UTD system won with just over 82%, well ahead of the next entries. Performance rose with more training data, and richer semantic resources tended to help modestly. The dataset matters because it offers a common, larger benchmark for relation classification, which supports applications such as information extraction and textual entailment.

Key points

  • Relation inventory: Cause-Effect, Instrument-Agency, Product-Producer, Content-Container, Entity-Origin, Entity-Destination, Component-Whole, Member-Collection and Message-Topic, plus OTHER. Arguments are common-noun heads in local syntactic contexts. Negated and modal cases are excluded.
  • Data split: 8,000 training and 2,717 test instances. The training set includes 891 re-annotated items from SemEval-1. Learning curves were reported for TD1–TD4 (1,000 / 2,000 / 4,000 / 8,000 examples).
  • Annotation agreement: inter-annotator agreement (exact match) ranged from about 58% (Entity-Origin) to 95.8% (Content-Container).
  • Results: UTD scored above 82%, FBK IRST 77.62% and ISI 77.57%. A random baseline reached about 6%, and a Naive Bayes baseline beat two systems. A majority vote of the top three systems gave 82.79%.
  • Per-relation and architecture findings: Cause-Effect was easiest, while Instrument-Agency and Product-Producer were hardest. Most systems used MaxEnt or SVM classifiers with WordNet, Google n-grams or Cyc. Building the task took over 1,000 person-hours.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

283 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 276 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hendrickx2010semeval,
  title = {{SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals}},
  author = {Hendrickx, Iris and Kim, Su Nam and Kozareva, Zornitsa and Nakov, Preslav and Séaghdha, Diarmuid Ó and Padó, Sebastian and Pennacchiotti, Marco and Romano, Lorenza and Szpakowicz, Stan},
  booktitle = {Proceedings of the 5th International Workshop on Semantic Evaluation},
  year = {2010},
  url = {https://aclanthology.org/S10-1006/}
}

References

5 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. X. Carreras and L. Màrquez. 2004. Introduction to the CoNLL-2004 shared task: Semantic role labeling. In Proc. CoNLL-04, Boston, MA. find
  2. I. Dagan, B. Dolan, B. Magnini, and D. Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches. Natural Language Engineering, 15(4):i–xvii. find
  3. R. Girju, P. Nakov, V. Nastase, S. Szpakowicz, P. Turney, and D. Yuret. 2009. Classification of semantic relations between nominals. Language Resources and Evaluation, 43(2):105–121. find
  4. I. Hendrickx, S. Kim, Z. Kozareva, P. Nakov, D. Ó Séaghdha, S. Padó, M. Pennacchiotti, L. Romano, and S. Szpakowicz. 2009. SemEval-2010 Task 8: Multi-way classification of semantic relations between pairs of nominals. In Proc. NAACL Workshop on Semantic Evaluations, Boulder, CO. find
  5. J. Kim, T. Ohta, S. Pyysalo, Y. Kano, and J. Tsujii. 2009. Overview of BioNLP’09 shared task on event extraction. In Proc. BioNLP-09, Boulder, CO. find