Elamur Research⌕
☰
home / papers / 2012 / SemEval-2012 Task 2: Measuring Degrees of Relational Similarity

SemEval-2012 Task 2: Measuring Degrees of Relational Similarity

Paper · 2012 · David Jurgens, Saif Mohammad, Peter Turney et al.

Summary

Most earlier work on semantic relations, including previous SemEval tasks, treats relations as discrete classes: a word pair either belongs to CAUSE : EFFECT or it does not. This task instead asks how typical a pair is of its relation. For example, dog:bark is a better example of ENTITY : SOUND than floor:squeak. Systems were given a few paradigm pairs and a schematic description for each relation. They then had to rank other pairs by how prototypical they are.

The organizers built the first graded relational-similarity dataset, covering 79 subcategories from the Bejar et al. (1991) taxonomy. It was collected with Amazon Mechanical Turk in two phases: Turkers first generated new example pairs, then judged how representative each pair was. Three teams submitted six systems, which were scored against the Turker rankings in two ways.

Results were modest overall. A simple PMI baseline beat every system except UTD-NB, and no approach did well across all relation types. The work matters for tasks that need graded judgments, such as ranked relational search and choosing the best source analogy.

Key points

  • Relation inventory: the taxonomy has 10 high-level categories (e.g., CAUSE-PURPOSE, SPACE-TIME) and 79 subcategories. Earlier relation tasks used only 7 or 9 relation types.
  • Data collection: Turkers produced 3160 pairs, of which 2905 were unique and 2823 remained after spelling cleanup. Another 395 pairs were added by reversing the arguments of sampled pairs, for 3218 rated pairs in total.
  • Quality control: roughly 7% of Turker responses were discarded because the worker chose the wrong relation description in a multiple-choice check.
  • Results (average Spearman's ρ with reversed pairs included): UTD-NB 0.229, UTD-SVM 0.116, PMI baseline 0.112, Duluth systems about 0.04–0.05, BUAP 0.014, random 0.018. Removing reversed pairs lowered the scores of the order-aware systems, which suggests those systems were correctly ranking reversals low.
  • Difficulty by category: systems did best on the Similar category and worst on Non-Attribute relations. Difficulty did not correlate with the spread of prototypicality scores within a subcategory.

Interesting ideas

  • I1. Defining a class by examples plus a schema. Each relation is specified only by a handful of seed examples and a short template such as "X is the goal of Y". This setup is useful wherever categories must be specified with almost no labeled data, for instance in low-resource Ukrainian annotation projects.
  • I2. A gate question before crowdsourced generation. Workers first pick the correct relation description from distractors, and only those who pass may contribute examples. This cheap filter removes confused or careless workers and could be reused in any crowdsourcing or volunteer annotation pipeline.
  • I3. Reversed pairs as built-in probes. Adding pairs with swapped arguments tests whether a model is sensitive to the direction of a relation. The authors also measure the rank gap between a pair and its reversal (RMSE) as a separate diagnostic, a trick that transfers to evaluating embeddings or knowledge-graph models.
  • I4. Testing a simple association baseline. PMI did surprisingly well because every pair within a subcategory is already related. Including such a baseline exposes whether the task design accidentally rewards plain co-occurrence instead of relational understanding.
  • I5. Choosing a metric that allows ablation. Rank correlation lets single items be removed for post-hoc analysis without disturbing the rest of the evaluation, whereas grouped comparison questions do not. This is worth considering when deciding how an evaluation should be structured.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️National Research Council CanadaOttawa · Canada40 works · 20,943 citations🎓University of California, Los AngelesLos Angeles · United States54 works · 53,688 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works

Cite

Cite as David Jurgens, Saif Mohammad, Peter Turney and Keith Holyoak. 2012. SemEval-2012 Task 2: Measuring Degrees of Relational Similarity. *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{jurgens2012semeval,
  title = {{SemEval-2012 Task 2: Measuring Degrees of Relational Similarity}},
  author = {David Jurgens, Saif Mohammad, Peter Turney and Keith Holyoak},
  booktitle = {*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)},
  year = {2012},
  url = {https://aclanthology.org/S12-1047/}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Isaac I. Bejar, Roger Chaffin, and Susan E. Embretson. 1991. Cognitive and Psychometric Analysis of Analogical Problem Solving. Springer-Verlag. find
  2. Cristina Butnariu, Su Nam Kim, Preslav Nakov, Dair-muid Ó Séaghdha, Stan Szpakowicz, and Tony Veale. 2010. Semeval-2010 task 9: The interpretation of noun compounds using paraphrasing verbs and prepositions. In Proceedings of the 5th International Workshop on Semantic Evaluation (SemEval-2010), pa… find
  3. Michael J. Cafarella, Michele Banko, and Oren Etzioni. 2006. Relational web search. In WWW Conference. find
  4. Kenneth W. Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22–29. our card
  5. Dedre Gentner. 1983. Structure-mapping: A theoretical framework for analogy. Cognitive science, 7(2):155– 170. find
  6. Roxana Girju, Preslav Nakov, Vivi Nastase, Stan Szpakowicz, Peter Turney, and Deniz Yuret. 2007. SemEval-2007 task 04: Classification of semantic relations between nominals. In Proceedings of the 4th International Workshop on Semantic Evaluation (SemEval-2007). our card
  7. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International… our card
  8. Keith J. Holyoak. 2012. Analogy and relational reasoning. In Oxford handbook of thinking and reasoning, pages 234–259. Oxford University Press. find
  9. Jordan J. Louviere. 1991. Best-worst scaling: A model for the largest difference judgments. Working Paper. find
  10. Saif M. Mohammad and Peter D. Turney. 2010. Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In Proceedings of the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 26–34. Association for Computati… our card
  11. Preslav Nakov and Marti A. Hearst. 2008. Solving relational similarity problems using the web as a corpus. In Proceedings of ACL, volume 8. find
  12. Vivi Nastase and Stan Szpakowicz. 2003. Exploring noun-modifier semantic relations. In Fifth International Workshop on Computational Semantics (IWCS-5), pages 285–301. ACL Press Tilburg,, The Netherlands. find
  13. Vivi Nastase, Jelber Sayyad-Shirabad, Marina Sokolova, and Stan Szpakowicz. 2006. Learning noun-modifier semantic relations with corpus-based and wordnet-based features. In Proceedings of AAAI, volume 21, page 781. find
  14. Bryan Orme. 2009. Maxdiff analysis: Simple counting, individual-level logit, and hb. find
  15. Patrick Pantel and Marco Pennacchiotti. 2006. Espresso: leveraging generic patterns for automatically harvesting semantic relations. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 11… our card
  16. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254–263. Association for Computational Lingui… our card
  17. M. Stephens, M. Palakal, S. Mukhopadhyay, R. Raje, J. Mostafa, et al. 2001. Detecting gene relations from medline abstracts. In Pacific Symposium on Biocomputing, volume 6, pages 483–495. Citeseer. find
  18. Peter D. Turney and Michael L Littman. 2005. Corpus-based learning of analogies and semantic relations. Machine Learning, 60(1–3):251–278. find
  19. Peter D. Turney. 2001. Mining the Web for synonyms: PMI-IR versus LSA on TOEFL. In Proceedings of the Twelfth European Conference on Machine Learning (ECML-2001), pages 491–502. find
  20. Peter D. Turney. 2005. Measuring semantic similarity by latent relational analysis. In Proceedings of IJCAI, pages 1136–1141. find
  21. Peter D. Turney. 2006. Similarity of semantic relations. Computational Linguistics, 32(3):379–416. our card
  22. Peter D. Turney. 2008. The latent relation mapping engine: Algorithm and experiments. Journal of Artificial Intelligence Research, 33(1):615–655. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jurgens-2012-semeval-2012-task-2-measuring/