Elamur Research
☰
home / papers / 2015 / SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/J15-4004.pdf

Summary

Popular benchmarks for word-level semantic models, such as WordSim-353 and MEN, reward association (coffee–cup) rather than true similarity (cup–mug). They also fail to separate the two. Hill, Reichart and Korhonen built SimLex-999 to fix this. It has 999 word pairs rated by 500 native English speakers recruited through Amazon Mechanical Turk, and the raters were told to judge similarity, not relatedness. The pairs were chosen systematically: they include adjectives, nouns and verbs from very concrete to very abstract. Each pair also carries a concreteness score and a free-association strength taken from the USF norms. The authors tested several model types: the neural language models of Huang et al., Collobert & Weston and Mikolov et al., plus count-based VSMs with and without SVD. On the older benchmarks these models already match human agreement, but on SimLex-999 they score well below it. Most of the errors come from pairs that are strongly associated but not similar. This leaves real room for progress and lets researchers analyse results by word type.

Key points

  • Association-heavy pairs score high in older benchmarks. For example, Maradona–football gets 8.62 in WordSim-353, while SimLex-999 gives such pairs low scores.
  • Each pair comes with POS, concreteness and USF association metadata, so results can be broken down into subsets (adjective/noun/verb, abstract/concrete, high/low association).
  • Models trained on dependency-parsed input separate similarity from association better than models trained on plain running text. Smaller context windows did not reliably help: the best size depended on the architecture, POS and concreteness.
  • The Mikolov et al. embeddings did best on every POS category. However, the count-based VSM and SVD models (window 10) beat them on the most concrete pairs, which suggests NLMs handle abstract meaning better.
  • The authors argue that models trained only on text may never reach human level, because much knowledge about concrete concepts comes from perception.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

447 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 432 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00237 · arXiv 1408.3456

@misc{hill2015simlex,
  title = {{SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation}},
  author = {Hill, Felix and Reichart, Roi and Korhonen, Anna},
  year = {2015},
  doi = {10.1162/coli\_a\_00237},
  eprint = {1408.3456},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/coli_a_00237}
}

References

76 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agirre, Eneko, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and Wordnet-based approaches. In Proceedings of NAACL, Boulder, CO. find
  2. Alfonseca, Enrique and Suresh Manandhar. 2002. Extending a lexical ontology by a combination of distributional semantics signatures. In G. Schrieber et al., Knowledge Engineering and Knowledge Management: Ontologies and the Semantic Web. Springer, pages 1–7. find
  3. Andrews, Mark, Gabriella Vigliocco, and David Vinson. 2009. Integrating experiential and distributional data to learn semantic representations. Psychological Review, 116(3):463. find
  4. Bansal, Mohit, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of ACL, Baltimore, MD. find our card
  5. Baroni, Marco, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of ACL, Baltimore, MD. find our card
  6. Baroni, Marco and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721. find our card
  7. Barsalou, Lawrence W., W. Kyle Simmons, Aron K. Barbey, and Christine D. Wilson. 2003. Grounding conceptual knowledge in modality-specific systems. Trends in Cognitive Sciences, 7(2):84–91. find
  8. Beltagy, Islam, Katrin Erk, and Raymond Mooney. 2014. Semantic parsing using distributional semantics and probabilistic logic. In ACL 2014 Workshop on Semantic Parsing. find
  9. Bengio, Yoshua, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137–1155. find
  10. Bernardi, Raffaella, Georgiana Dinu, Marco Marelli, and Marco Baroni. 2013. A relatedness benchmark to test the role of determiners in compositional distributional semantics. In Proceedings of ACL, Sofia. find
  11. Biemann, Chris. 2005. Ontology learning from text: A survey of methods. LDV Forum, 20(2):75–93. find
  12. Bird, Steven. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL on Interactive Presentation sessions, pages 69–72, Sydney. find
  13. Bruni, Elia, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012a. Distributional semantics in technicolor. In Proceedings of ACL, Jeju Island. find
  14. Bruni, Elia, Jasper Uijlings, Marco Baroni, and Nicu Sebe. 2012b. Distributional semantics with eyes: Using image analysis to improve computational representations of word meaning. In Proceedings of the 20th ACM International Conference on Multimedia, Nara. find
  15. Budanitsky, Alexander and Graeme Hirst. 2006. Evaluating Wordnet-based measures of lexical semantic relatedness. Computational Linguistics, 32(1):13–47. find our card
  16. Cimiano, Philipp, Andreas Hotho, and Steffen Staab. 2005. Learning concept hierarchies from text corpora using formal concept analysis. J. Artif. Intell. Res. (JAIR), 24:305–339. find
  17. Collobert, R. and J. Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In International Conference on Machine Learning, ICML, Helsinki. find
  18. Cruse, D. Alan. 1986. Lexical semantics. Cambridge University Press. find
  19. Cunningham, Hamish. 2005. Information extraction, automatic. Encyclopedia of language and linguistics, pages 665–677. find
  20. Fellbaum, Christiane 1998. WordNet. Wiley Online Library. find
  21. Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th International Conference on World Wide Web, pages 406–414, Hong Kong. find
  22. Firth, J. R. 1957. Papers in Linguistics 1934–1951. Oxford University Press. find
  23. Gentner, Dedre. 1978. On relational meaning: The acquisition of verb meaning. Child Development, pages 988–998. find
  24. Gentner, Dedre. 2006. Why verbs are hard to learn. Action meets word: How Children Learn Verbs, pages 544–564. find
  25. Gershman, Anatole, Yulia Tsvetkov, Leonid Boytsov, Eric Nyberg, and Chris Dyer. 2014. Metaphor detection with cross-lingual model transfer. In Proceedings of ACL, Baltimore, MD. find
  26. Golub, Gene H. and Christian Reinsch. 1970. Singular value decomposition and least squares solutions. Numerische Mathematik, 14(5):403–420. find
  27. Griffiths, Thomas L., Mark Steyvers, and Joshua B. Tenenbaum. 2007. Topics in semantic representation. Psychological Review, 114(2):211. find
  28. Haghighi, Aria, Percy Liang, Taylor Berg-Kirkpatrick, and Dan Klein. 2008. Learning bilingual lexicons from monolingual corpora. In Proceedings of ACL 2008, Columbus, OH. find
  29. Hassan, Samer and Rada Mihalcea. 2011. Semantic relatedness using salient semantic analysis. In AAAI, San Francisco, CA. find
  30. Hatzivassiloglou, Vasileios, Judith L. Klavans, Melissa L. Holcombe, Regina Barzilay, Min-Yen Kan, and Kathleen McKeown. 2001. Simfinder: A flexible clustering tool for summarization. In NAACL Workshop on Automatic Summarization, Pittsburgh, PA. find
  31. He, Xiaodong, Mei Yang, Jianfeng Gao, Patrick Nguyen, and Robert Moore. 2008. Indirect-HMM-based hypothesis alignment for combining outputs from machine translation systems. In Proceedings of EMNLP, pages 98–107, Edinburgh. find
  32. Hill, Felix, Douwe Kiela, and Anna Korhonen. 2013. Concreteness and corpora: A theoretical and practical analysis. find
  33. Hill, Felix, Anna Korhonen, and Christian Bentz. 2014. A quantitative empirical analysis of the abstract/concrete distinction. Cognitive Science, 38(1):162–177. find
  34. Hill, Felix, Roi Reichart, and Anna Korhonen. 2014. Multi-modal models for concrete and abstract concept meaning. Transactions of the Association for Computational Linguistics (TACL), 2:285–296. find
  35. Huang, Eric H., Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of ACL, pages 873–882, Jeju Island. find
  36. Kiela, Douwe and Stephen Clark. 2014. A systematic study of semantic vector space model parameters. In Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC)@ EACL, pages 21–30, Gothenburg. find
  37. Kiela, Douwe, Felix Hill, Anna Korhonen, and Stephen Clark. 2014. Improving multi-modal representations using image dispersion: Why less is sometimes more. In Proceedings of ACL, Baltimore, MD. find
  38. Landauer, Thomas K. and Susan T. Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211. find
  39. Leech, Geoffrey, Roger Garside, and Michael Bryant. 1994. Claws4: The tagging of the British National Corpus. In Proceedings of COLING, pages 622–628, Kyoto. find
  40. Levy, Omer and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of ACL, volume 2. find
  41. Levy, Omer, Steffen Remus, Chris Biemann, and Idol Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? Proceedings of NAACL, Denver, CO. find our card
  42. Lewis, David D., Yiming Yang, Tony G. Rose, and Fan Li. 2004. Rcv1: A new benchmark collection for text categorization research. The Journal of Machine Learning Research, 5:361–397. find our card
  43. Li, Changliang, Bo Xu, Gaowei Wu, Xiuying Wang, Wendong Ge, and Yan Li. 2014. Obtaining better word representations via language transfer. In A. Gelbukh, editor, Computational Linguistics and Intelligent Text Processing. Springer, pages 128–137. find
  44. Li, Mu, Yang Zhang, Muhua Zhu, and Ming Zhou. 2006. Exploring distributional similarity based models for query spelling correction. In Proceedings of ALC, pages 1025–1032. find
  45. Luong, Minh-Thang, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. CoNLL-2013, page 104, Sofia. find our card
  46. Markman, Arthur B. and Edward J. Wisniewski. 1997. Similar and different: The differentiation of basic-level categories. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23(1):54. find
  47. Marton, Yuval, Chris Callison-Burch, and Philip Resnik. 2009. Improved statistical machine translation using monolingually-derived paraphrases. In Proceedings of EMNLP, pages 381–390, Edinburgh. find
  48. McRae, Ken, Saman Khalkhali, and Mary Hare. 2012. Semantic and associative relations in adolescents and young adults: Examining a tenuous dichotomy. In Valerie F. Reyna, Sandra B. Chapman, Michael R. Dougherty, and Jere Ed Confrey, editors, The Adolescent Brain: Learning, Reasoning, and Decision Mak… find
  49. Medelyan, Olena, David Milne, Catherine Legg, and Ian H. Witten. 2009. Mining meaning from Wikipedia. International Journal of Human-Computer Studies, 67(9):716–754. find
  50. Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of International Conference of Learning Representations, Scottsdale, AZ. find
  51. Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119, Lake Tahoe, NV. find
  52. Navigli, Roberto. 2009. Word sense disambiguation: A survey. ACM Computing Surveys (CSUR), 41(2):10. find
  53. Nelson, Douglas L., Cathy L. McEvoy, and Thomas A. Schreiber. 2004. The University of South Florida free association, rhyme, and word fragment norms. Behavior Research Methods, Instruments, & Computers, 36(3):402–407. find
  54. Padó, Sebastian, Ulrike Padó, and Katrin Erk. 2007. Flexible, corpus-based modelling of human plausibility judgements. In Proceedings of EMNLP-CoNLL, pages 400–409, Prague. find
  55. Paivio, Allan. 1991. Dual coding theory: Retrospect and current status. Canadian Journal of Psychology/Revue canadienne de psychologie, 45(3):255. find
  56. Pedersen, Ted, Siddharth Patwardhan, and Jason Michelizzi. 2004. Wordnet:: Similarity: Measuring the relatedness of concepts. In Demonstration Papers at HLT-NAACL 2004, pages 38–41, New York, NY. find
  57. Phan, Xuan-Hieu, Le-Minh Nguyen, and Susumu Horiguchi. 2008. Learning to classify short and sparse text & Web with hidden topics from large-scale data collections. In Proceedings of the 17th International Conference on World Wide Web, pages 91–100, Beijing. find
  58. Plaut, David C. 1995. Semantic and associative priming in a distributed attractor network. In Proceedings of CogSci, volume 17, pages 37–42, Pittsburgh, PA. find
  59. Recchia, Gabriel and Michael N. Jones. 2009. More data trumps smarter algorithms: Comparing pointwise mutual information with latent semantic analysis. Behavior Research Methods, 41(3):647–656. find
  60. Reisinger, Joseph and Raymond Mooney. 2010a. A mixture model with sharing for lexical semantics. In Proceedings of EMNLP, pages 1173–1182, Cambridge, MA. find
  61. Reisinger, Joseph and Raymond J. Mooney. 2010b. Multi-prototype vector-space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 109–117, Los Angeles, CA. find our card
  62. Resnik, Philip. 1995. Using information content to evaluate semantic similarity in a taxonomy. In Proceedings of IJCAI. find our card
  63. Resnik, Philip and Jimmy Lin. 2010. 11 evaluations of NLP systems. The handbook of computational linguistics and natural language processing, 57:271. find
  64. Rosch, Eleanor, Carol Simpson, and R. Scott Miller. 1976. Structural bases of typicality effects. Journal of Experimental Psychology: Human Perception and Performance, 2(4):491. find
  65. Rose, Tony, Mark Stevenson, and Miles Whitehead. 2002. The Reuters corpus volume 1—from yesterday’s news to tomorrow’s language resources. In LREC, volume 2, pages 827–832, Las Palmas. find
  66. Rubenstein, Herbert and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. find
  67. Silberer, Carina and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In Proceedings of ACL, Sofia. find
  68. Sun, Lin, Anna Korhonen, and Yuval Krymolowski. 2008. Verb class discovery from rich syntactic data. In A. Gelbukh, editor, Computational Linguistics and Intelligent Text processing. Springer, pages 16–27. find
  69. Turian, Joseph, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of ACL, pages 384–394, Uppsala. find our card
  70. Turney, Peter D. 2012. Domain and function: A dual-space model of semantic relations and compositions. Journal of Artificial Intelligence Research (JAIR), 179(44):533–585. find
  71. Turney, Peter D. and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37(1):141–188. find
  72. Tversky, Amos. 1977. Features of similarity. Psychological Review, 84(4):327. find
  73. Wiebe, Janyce. 2000. Learning subjective adjectives from corpora. In AAAI/IAAI, pages 735–740, Austin, TX. find
  74. Williams, Gbolahan K. and Sarabjot Singh Anand. 2009. Predicting the polarity strength of adjectives using Wordnet. In ICWSM, San Jose, CA. find
  75. Wu, Zhibiao and Martha Palmer. 1994. Verbs, semantics and lexical selection. In Proceedings of ACL, pages 133–138, Las Cruces, NM. find
  76. Yong, Chung and Shou King Foo. 1999. A case study on inter-annotator agreement for word sense disambiguation. In Proceedings of the ACL SIGLEX Workshop on Standardizing Lexical Resources (SIGLEX99), College Park, MD. find