Elamur Research⌕
☰
home / papers / 2016 / SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity

SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity

Summary

Verbs are central to sentence meaning, yet intrinsic benchmarks for word representations have mostly targeted nouns or included only a few verb pairs. On the existing verb subset of SimLex-999, automatic systems already score above inter-annotator agreement. To address this, the authors built SimVerb-3500, a set of 3,500 verb pairs with human similarity ratings on a 0–10 scale. Annotators followed the SimLex-999 guidelines, which separate similarity from relatedness and ask that antonyms receive low scores. Pairs were sampled so the set covers every normed verb in the USF free-association norms and includes several members of each top-level VerbNet class. This ties the dataset to resources such as WordNet, PropBank and FrameNet. The authors benchmarked several representation models, including SGNS with bag-of-words and dependency contexts, symmetric-pattern vectors, non-distributional vectors, Paragram and Paragram+CF. These systems remain clearly below human agreement. Raw-text distributional models perform especially poorly on rare and highly polysemous verbs. The resource makes statistically robust, fine-grained comparisons of verb representations possible.

Key points

  • SimVerb-3500 has 3,500 rated pairs covering 827 verb types from USF, and every one of the 101 top-level VerbNet classes is represented by no fewer than 3 verbs.
  • SimLex-999 has only 222 verb pairs and 170 verb lemmas, and it leaves 39 top-level VerbNet classes uncovered. On that subset, the best system (ρ = 0.727, Mrkšić et al., 2016) slightly exceeds human agreement (ρ = 0.717).
  • Each pair carries a WordNet-derived relation label. Examples include 373 no-relation pairs, 306 synonym pairs and 800 hyper/hyponym pairs.
  • Model scores differ noticeably across VerbNet classes, for example psychological-state verbs (class 31) versus motion verbs (class 51). Paragram+CF is the strongest model overall in these breakdowns.
  • Annotator rating spread is about 1.3 regardless of verb frequency or polysemy. It is lower for antonyms (0.86) and unrelated pairs (0.92), which suggests that humans, unlike models, are largely unaffected by frequency and ambiguity.

Interesting ideas

  • I1. Sample pairs from association norms. Drawing pairs from free-association data yields both truly similar and merely related pairs. This forces a benchmark to test whether models can tell the two apart, and it would work for any language with association norms.
  • I2. Anchor coverage in a lexical taxonomy. Guaranteeing a minimum number of items per class of a lexicon such as VerbNet makes the benchmark representative by design. It also enables per-class error analysis and links the data to other linked resources.
  • I3. Report results by subset. Breaking scores down by frequency band, polysemy, lexical relation and semantic class shows where a model fails instead of hiding weaknesses in one global number. The same practice suits treebank or LAS evaluations.
  • I4. Use rating variance as a diagnostic. Comparing the average standard deviation of human ratings across subsets separates genuinely hard items from items that only models find hard.
  • I5. Check whether the metric fits each subset. Spearman's ρ is unsuitable for groups like antonyms, whose internal ranking is arbitrary because all should score near zero. Verifying that a rank-based metric suits each subset before reporting it is a useful general habit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Word sense disambiguation · Semantic role labeling

Cited by

97 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 91 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1235

@inproceedings{gerz2016simverb,
  title = {{SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity}},
  author = {Gerz, Daniela and Vulić, Ivan and Hill, Felix and Reichart, Roi and Korhonen, Anna},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1235},
  url = {https://doi.org/10.18653/v1/d16-1235}
}

References

36 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Enrique Alfonseca, Keith B. Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In NAACL-HLT, pages 19–27. find our card
  2. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In ACL-COLING, pages 86–90. find our card
  3. Simon Baker, Roi Reichart, and Anna Korhonen. 2014. An unsupervised model for instance level subcategorization acquisition. In EMNLP, pages 278–289. find
  4. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL, pages 238–247. find our card
  5. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1–47. find
  6. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML, pages 160–167. find
  7. Manaal Faruqui and Chris Dyer. 2015. Non-distributional word vector representations. In ACL, pages 464–469. find
  8. Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard H. Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In NAACL-HLT, pages 1606–1615. find
  9. Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. 2016. Problems with evaluation of word embeddings using word similarity tasks. CoRR, abs/1605.02276. find our card
  10. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1):116– 131. find
  11. Jeffrey Gruber. 1976. Lexical structure in syntax and semantics. North-Holland Pub. Co. find
  12. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. find our card
  13. Ray S. Jackendoff. 1972. Semantic interpretation in generative grammar. find
  14. MIT Press. Adam Kilgarriff. 1997. Putting frequencies in the dictionary. International Journal of Lexicography, 10(2):135– 155. find
  15. Karin Kipper, Benjamin Snyder, and Martha Palmer. 2004. Extending a verb-lexicon using a semantically annotated corpus. In LREC, pages 1557–1560. find
  16. Karin Kipper, Anna Korhonen, Neville Ryant, and Martha Palmer. 2008. A large-scale classification of English verbs. Language Resource and Evaluation, 42(1):21– 40. find
  17. Anna Korhonen. 2010. Automatic lexical classification: bridging research and practice. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 368(1924):3621– 3632. find
  18. Brenden M. Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B. Tenenbaum. 2011. One shot learning of simple visual concepts. In CogSci. find
  19. Beth Levin. 1993. English verb classes and alternation, A preliminary investigation. The University of Chicago Press. find
  20. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In ACL, pages 302–308. find
  21. Edward Loper, Szu-Ting Yi, and Martha Palmer. 2007. Combining lexical resources: Mapping between Prop-Bank and VerbNet. In IWCS. find
  22. Thang Luong, Richard Socher, and Christopher Manning. 2013. Better word representations with recursive neural networks for morphology. In CoNLL, pages 104–113. find our card
  23. Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In ICLR: Workshop Papers. find
  24. George A. Miller. 1995. Wordnet: A lexical database for english. Communications of the ACM, 38(11):39–41. Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina Maria Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve J. Young. 2016. Counter-fitting word vectors… find our card
  25. Douglas L. Nelson, Cathy L. McEvoy, and Thomas A. Schreiber. 2004. The University of South Florida free association, rhyme, and word fragment norms. Behavior Research Methods, Instruments, & Computers, 36(3):402–407. find
  26. Sebastian Padó, Ulrike Padó, and Katrin Erk. 2007. Flexible, corpus-based modelling of human plausibility judgements. In EMNLP-CoNLL, pages 400–409. find
  27. Martha Palmer, Paul Kingsbury, and Daniel Gildea. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. find our card
  28. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In EMNLP, pages 1532–1543. find
  29. Joseph Reisinger and Raymond J. Mooney. 2010a. A mixture model with sharing for lexical semantics. In EMNLP, pages 1173–1182. find
  30. Joseph Reisinger and Raymond J Mooney. 2010b. Multi-prototype vector-space models of word meaning. In NAACL-HTL, pages 109–117. find our card
  31. Herbert Rubenstein and John B Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  32. Roy Schwartz, Roi Reichart, and Ari Rappoport. 2015. Symmetric pattern based word embeddings for improved word similarity prediction. In CoNLL, pages 258–267. find
  33. Carina Silberer and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In ACL, pages 721–732. find
  34. Joseph P. Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In ACL, pages 384–394. find our card
  35. Luke Vilnis and Andrew McCallum. 2015. Word representations via Gaussian embedding. find
  36. ICLR. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. From paraphrase database to compositional paraphrase model and back. Transactions of the ACL, 3:345–358. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gerz-2016-simverb-3500-large-scale-evaluation/