Elamur Research⌕
☰
home / papers / 2017 / HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment

HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment

Paper · 2017 · Ivan Vulić, Daniela Gerz, Douwe Kiela et al.

Summary

Most NLP resources, such as WordNet, encode the hyponymy–hypernymy (type-of) relation as a yes/no fact. Cognitive psychology, however, shows that people treat category membership and typicality as matters of degree. HyperLex addresses this mismatch with an English benchmark for graded lexical entailment (LE).

The authors used crowdsourcing to collect ratings from native English speakers. Annotators scored how strongly one concept is a kind of another for 2,616 noun and verb pairs. The pairs were chosen to vary in part of speech, concreteness, hierarchy level and WordNet relation.

The ratings are highly consistent across annotators. They also reproduce Rosch-style prototype effects and reflect the directional, asymmetric nature of LE. True hypernym pairs receive the highest scores, which suggests the data refine WordNet rather than contradict it.

The authors then tested a broad set of systems against the human scores:

  • unsupervised distributional models
  • multimodal models
  • WordNet-based models
  • embedding-based models
  • supervised models

All of them fall far below human agreement. HyperLex therefore serves as a demanding testbed for semantic models that are meant to handle degrees of category membership.

Key points

  • Size and rating coverage: The dataset has 2,616 concept pairs (nouns and verbs). Each pair was scored by at least 10 raters, drawn from a pool of more than 600 native English annotators.
  • Agreement: Inter-annotator agreement reaches roughly 0.85 Spearman's ρ, both pairwise and mean. Agreement is similarly high on the noun and verb subsets taken separately.
  • Evaluated approaches:
    • distributional inclusion and generality measures (e.g., Santus et al. 2014)
    • multimodal models (Kiela et al. 2015)
    • WordNet-based similarity measures
    • word embeddings, including the LE-oriented models of Vilnis and McCallum (2015) and Vendrov et al. (2016)
    • supervised LE classifiers, such as those proposed by Weeds et al. (2014), by Roller, Erk, and Boleda (2014), and by Baroni et al. (2012)
  • Supervised use: The release provides standard train/dev/test splits, in both a random and a lexical (vocabulary-disjoint) version.
  • Availability: HyperLex is freely available from the authors' Cambridge LTL webpage. The article itself is published under the CC BY-NC-ND 4.0 license.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States158 works · 211,459 citations🎓University of CambridgeCambridge · United Kingdom155 works · 115,562 citations🏢Google DeepMindLondon · United Kingdom35 works · 75,886 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works

Cite

Cite as Ivan Vulić, Daniela Gerz, Douwe Kiela, Felix Hill and Anna Korhonen. 2017. HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00301

@article{vulic2017hyperlex,
  title = {{HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment}},
  author = {Ivan Vulić, Daniela Gerz, Douwe Kiela, Felix Hill and Anna Korhonen},
  journal = {Computational Linguistics},
  year = {2017},
  doi = {10.1162/coli\_a\_00301},
  url = {https://doi.org/10.1162/coli_a_00301}
}

References

165 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agić, Željko and Natalie Schluter. 2017. Baselines and test data for cross-lingual inference. CoRR, abs/1704.05347. find
  2. Agirre, Eneko, Enrique Alfonseca, Keith B. Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of NAACL-HLT, pages 19–27, Boulder, CO. our card
  3. Al-Rfou, Rami, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. In Proceedings of CoNLL, pages 183–192, Sofia. our card
  4. Androutsopoulos, Ion and Prodromos Malakasiotis. 2010. A survey of paraphrasing and textual entailment methods. Journal of Artificial Intelligence Research, 38:135–187. our card
  5. Astudillo, Ramón, Silvio Amir, Wang Ling, Mario Silva, and Isabel Trancoso. 2015. Learning word representations from scarce and noisy data with embedding subspaces. In Proceedings of ACL, pages 1074–1084, Beijing. find
  6. Auer, Sören, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. 2007. DBPedia: A nucleus for a Web of open data. In Proceedings of the Semantic Web Conference (ISWC), pages 722–735, Busan. find
  7. Bankova, Desislava, Bob Coecke, Martha Lewis, and Daniel Marsden. 2016. Graded entailment for compositional distributional semantics. CoRR, abs/1601.04908. — [arXiv:1601.04908](https://arxiv.org/abs/1601.04908) find
  8. Baronett, Stan. 2012. Logic, 3rd edition. find
  9. Baroni, Marco, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012. Entailment above the word level in distributional semantics. In Proceedings of EACL, pages 23–32, Avignon. our card
  10. Baroni, Marco, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of ACL, pages 238–247, Baltimore, MD. our card
  11. Baroni, Marco and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721. our card
  12. Baroni, Marco and Alessandro Lenci. 2011. How we BLESSed distributional semantic evaluation. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics (GEMS), pages 1–10, Edinburgh. our card
  13. Batchkarov, Miroslav, Thomas Kober, Jeremy Reffin, Julie Weeds, and David Weir. 2016. A critique of word similarity as a method for evaluating distributional semantic models. In Proceedings of REPEVAL, pages 7–12, Berlin. find
  14. Beckwith, Richard, Christiane Fellbaum, Derek Gross, and George A. Miller. 1991. WordNet: A lexical database organized on psycholinguistic principles. Lexical acquisition: Exploiting on-line resources to build a lexicon, pages 211–231. find
  15. Bejar, Isaac I., Roger Chaffin, and Susan Embretson. 1991. Cognitive and Psychometric Analysis of Analogical Problem Solving. find
  16. Springer. Beltagy, Islam, Cuong Chau, Gemma Boleda, Dan Garrette, Katrin Erk, and Raymond Mooney. 2013. Montague meets Markov: Deep semantics with probabilistic logical form. In Proceedings of *SEM, pages 11–21, Atlanta, GA. van Benthem, Johan and Alice ter Meulen. 1996. Handbook of Logic and Langua… find
  17. Elsevier. Bernardi, Raffaella, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. 2016. Automatic description generation from images: A survey of models, datasets, and evaluation measures. Journal of Artificial Intelligence… find
  18. Biran, Or and Kathleen McKeown. 2013. Classifying taxonomic relations between pairs of Wikipedia articles. In Proceedings of IJCNLP, pages 788–794, Nagoya. find
  19. Blutner, Reinhard, Emmanuel M. Pothos, and Peter Bruza. 2013. A quantum probability perspective on borderline vagueness. Topics in Cognitive Science, 5(4):711–736. find
  20. Bordes, Antoine, Jason Weston, Ronan Collobert, and Yoshua Bengio. 2011. Learning structured embeddings of knowledge bases. In Proceedings of AAAI, pages 301–306, San Francisco, CA. find
  21. Bos, Johan and Katja Markert. 2005. Recognising textual entailment with logical inference. In Proceedings of EMNLP, pages 628–635, Vancouver. our card
  22. Bowman, Samuel R., Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP, pages 632–642, Lisbon. find
  23. Bruni, Elia, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1–47. find
  24. Bullinaria, John A. and Joseph P. Levy. 2007. Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior Research Methods, 39(3):510–526. find
  25. Cafarella, M. J., Banko, M., & Etzioni, O. 2006. Relational Web search. Technical Report 2006-04-02, University of Washington, Department of Computer Science and Engineering, Edinburgh. find
  26. Camacho-Collados, José, Mohammad Taher Pilehvar, and Roberto Navigli. 2015. A framework for the construction of monolingual and cross-lingual word similarity datasets. In Proceedings of ACL, pages 1–7, Beijing. find
  27. Clarke, Daoud. 2009. Context-theoretic semantics for natural language: An Overview. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics (GEMS), pages 112–119, Athens. find
  28. Coleman, Linda and Paul Kay. 1981. Prototype semantics: The English word lie. Language, 57(1):26–44. find
  29. Collins, Allan M. and Ross M. Quillian. 1969. Retrieval time from semantic memory. Journal of Verbal Learning and Verbal Behavior, 8(2):240–247. find
  30. Collins, Allan M. and Ross M. Quillian. 1972. Experiments on semantic memory and language comprehension. Cognition in Learning and Memory. find
  31. Dagan, Ido, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, pages 177–190. find
  32. Dagan, Ido, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4):1–220. find
  33. Decock, Lieven and Igor Douven. 2014. What is graded membership? Noûs, 48(4):653–682. find
  34. Deng, Jia, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of CVPR, pages 248–255, Miami, FL. find
  35. Deselaers, Thomas and Vittorio Ferrari. 2011. Visual and semantic similarity in ImageNet. In Proceedings of CVPR, pages 1777–1784, Colorado Springs, CO. find
  36. Dirven, René and John Taylor. 1986. The conceptualisation of vertical space in English: The case of tall. Linguistic Agency, University of Duisburg. find
  37. Divjak, Dagmar and Antti Arppe. 2013. Extracting prototypes from exemplars: What can corpus data tell us about concept representation? Cognitive Linguistics, 24(2):221–274. find
  38. Do, Quang and Dan Roth. 2010. Constraints based taxonomic relation classification. In Proceedings of EMNLP, pages 1099–1109, Cambridge, MA. find
  39. Esteva, Francesc, Lluís Godo, Ricardo O. Rodríguez, and Thomas Vetterlein. 2012. Logics for approximate and strong entailments. Fuzzy Sets and Systems, 197:59–70. find
  40. Evert, Stefan. 2008. Corpora and collocations. Corpus Linguistics, 2:223–233. find
  41. Faruqui, Manaal, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In Proceedings of NAACL-HLT, pages 1606–1615, Denver, CO. find
  42. Faruqui, Manaal and Chris Dyer. 2015. Non-distributional word vector representations. In Proceedings of ACL, pages 464–469, Beijing. find
  43. Faruqui, Manaal, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. 2016. Problems with evaluation of word embeddings using word similarity tasks. In Proceedings of REPEVAL, pages 30–35, Berlin. our card
  44. Fellbaum, Christiane. 1998. find
  45. WordNet. Cambridge, MA, MIT Press. Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1):116–131. find
  46. Fromkin, Victoria, Robert Rodman, and Nina Hyams. 2013. An Introduction to Language, 10th edition. find
  47. Wadsworth Cengage Learning. Fu, Ruiji, Jiang Guo, Bing Qin, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014. Learning semantic hierarchies via word embeddings. In Proceedings of ACL, pages 1199–1209, Baltimore, MD. our card
  48. Fu, Ruiji, Jiang Guo, Bing Qin, Wanxiang Che, Haifeng Wang, and Ting Liu. 2015. Learning semantic hierarchies: A continuous vector space approach. IEEE/ACM Transactions on Audio, Speech & Language Processing, 23(3): 461–471. find
  49. Gärdenfors, Peter. 2004. Conceptual Spaces: The Geometry of Thought. find
  50. MIT Press. Geffet, Maayan and Ido Dagan. 2005. The distributional inclusion hypotheses and lexical entailment. In Proceedings of ACL, pages 107–114, Ann Arbor, MI. our card
  51. Gentner, Dedre. 1983. Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2):155–170. find
  52. Gentner, Dedre. 2006. Why verbs are hard to learn. Action meets word: How children learn verbs, pages 544–564. Oxford University Press. Gerz, Daniela, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. SimVerb-3500: A large-scale evaluation set of verb similarity. In Proceedings of EMNL… find
  53. Gheorghita, Inga and Jean-Marie Pierrel. 2012. Towards a methodology for automatic identification of hypernyms in the definitions of large-scale dictionary. In Proceedings of LREC, pages 2614–2618, Istanbul. find
  54. Gladkova, Anna and Aleksandr Drozd. 2016. Intrinsic evaluations of word embeddings: What can we do better? In Proceedings of REPEVAL, pages 36–42, Berlin. find
  55. Gulordava, Kristina and Marco Baroni. 2011. A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. In Proceedings of the GEMS Workshop on GEometrical Models of Natural Language Semantics, pages 67–71, Edinburgh. our card
  56. Hampton, James A. 2006. Concepts as prototypes. Psychology of Learning and Motivation, 46:79–113. find
  57. Hampton, James A. 2007. Typicality, graded membership, and vagueness. Cognitive Science, 31(3):355–384. find
  58. Harris, Zellig S. 1954. Distributional structure. Word, 10(2-3):146–162. find
  59. Hashimoto, Tatsunori B., David Alvarez-Melis, and Tommi S. Jaakkola. 2016. Word embeddings as metric recovery in semantic spaces. Transactions of the ACL, 4:273–286. find
  60. He, Shizhu, Kang Liu, Guoliang Ji, and Jun Zhao. 2015. Learning to represent knowledge graphs with Gaussian embedding. In Proceedings of CIKM, pages 623–632, Melbourne. find
  61. Hearst, Marti A. 1992. Automatic acquisition of hyponyms from large text corpora. In Proceedings of COLING, pages 539–545, Nantes. find
  62. Hendrickx, Iris, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of SEMEVAL, pages 33–38, U… our card
  63. Herbelot, Aurélie and Mohan Ganesalingam. 2013. Measuring semantic content in distributional vectors. In Proceedings of ACL, pages 440–445, Sofia. find
  64. Hill, Felix, Anna Korhonen, and Christian Bentz. 2014. A quantitative empirical analysis of the abstract/concrete distinction. Cognitive Science, 38(1):162–177. find
  65. Hill, Felix, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. our card
  66. Hodosh, Micah, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853–899. find
  67. Holyoak, Keith J. 2012. Analogy and relational reasoning. In K. J. Holyoak and R. G. Morrison, editors, The Oxford Handbook of Thinking and Reasoning. Oxford University Press, pages 234–259. find
  68. Jackendoff, Ray S. 1972. Semantic Interpretation in Generative Grammar. MIT Press . Jauhar, Sujay Kumar, Chris Dyer, and Eduard Hovy. 2015. Ontologically grounded multi-sense representation learning for semantic vector space models. In Proceedings of NAACL-HLT, pages 683–693, Denver, CO. find
  69. Jia, Yangqing, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross B. Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of ACM Multimedia, pages 675–678, Orlando, FL. our card
  70. Jurgens, David, Saif Mohammad, Peter Turney, and Keith Holyoak. 2012. SemEval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of SEMEVAL, pages 356–364, Montreal. our card
  71. Kamp, Hans and Barbara Partee. 1995. Prototype theory and compositionality. Cognition, 57(2):129–191. find
  72. Kato, Makoto P., Hiroaki Ohshima, Satoshi Oyama, and Katsumi Tanaka. 2009. Query by analogical example: Relational search using Web search engine indices. In Proceedings of CIKM, pages 27–36, Hong Kong. find
  73. Kiela, Douwe and Léon Bottou. 2014. Learning image embeddings using convolutional neural networks for improved multi-modal semantics. In Proceedings of EMNLP, pages 36–45, Doha. find
  74. Kiela, Douwe, Felix Hill, and Stephen Clark. 2015. Specializing word embeddings for similarity or relatedness. In Proceedings of EMNLP, pages 2044–2048, Lisbon. find
  75. Kiela, Douwe, Felix Hill, Anna Korhonen, and Stephen Clark. 2014. Improving multi-modal representations using image dispersion: Why less is sometimes more. In Proceedings of ACL, pages 835–841, Baltimore, MD. Kiela, Douwe, Laura Rimell, Ivan Vulić, and Stephen Clark. 2015. Exploiting image generali… find
  76. Kipper, Karin, Anna Korhonen, Neville Ryant, and Martha Palmer. 2008. A large-scale classification of English verbs. Language Resources and Evaluation, 42(1):21–40. find
  77. Kopliku, Arlind, Karen Pinel-Sauvagnat, and Mohand Boughanem. 2011. Retrieving attributes using Web tables. In Proceedings of JCDL, pages 397–398, Ottawa. find
  78. Korhonen, Anna. 2010. Automatic lexical classification: bridging research and practice. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 368(1924):3621–3632. find
  79. Kotlerman, Lili, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geffet. 2010. Directional distributional similarity for lexical inference. Natural Language Engineering, 16(4):359–389. find
  80. Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of NIPS, pages 1106–1114, South Lake Tahoe, NV. find
  81. Lake, Brenden M., Ruslan Salakhutdinov, Jason Gross, and Joshua B. Tenenbaum. 2011. One shot learning of simple visual concepts. In Proceedings of CogSci, pages 2568–2573, Boston, MA. find
  82. Lakoff, George. 1990. Women, Fire, and Dangerous Things: What Categories Reveal about the Mind. University of Chicago Press. find
  83. Lazaridou, Angeliki, Nghia The Pham, and Marco Baroni. 2015. Combining language and vision with a multimodal skip-gram model. In Proceedings of NAACL-HLT, pages 153–163, Denver, CO. our card
  84. Leacock, Claudia and Martin Chodorow. 1998. Combining local context and WordNet similarity for word sense identification. WordNet: An electronic lexical database, 49(2):265–283. find
  85. Lenci, Alessandro and Giulia Benotto. 2012. Identifying hypernyms in distributional semantic spaces. In Proceedings of *SEM, pages 75–79, Montreal. our card
  86. Leviant, Ira and Roi Reichart. 2015. Separated by an un-common language: Towards judgment language informed vector space modeling. CoRR, abs/1508.00106. — [arXiv:1508.00106](https://arxiv.org/abs/1508.00106) find
  87. Levin, Beth. 1993. English Verb Classes and Alternation: A Preliminary Investigation. University of Chicago Press. find
  88. Levy, Omer, Ido Dagan, and Jacob Goldberger. 2014. Focused entailment graphs for open IE propositions. In Proceedings of CoNLL, pages 87–97, Baltimore, MD. find
  89. Levy, Omer and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of ACL, pages 302–308, Baltimore, MD. find
  90. Levy, Omer, Steffen Remus, Chris Biemann, and Ido Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? In Proceedings of NAACL-HLT, pages 970–976, Denver, CO. our card
  91. Lin, Dekang. 1998. Automatic retrieval and clustering of similar words. In Proceedings of ACL, pages 768–774, Montreal. our card
  92. Lin, Yankai, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of AAAI, pages 2181–2187, Austin, TX. find
  93. Liu, Quan, Hui Jiang, Si Wei, Zhen-Hua Ling, and Yu Hu. 2015. Learning semantic word embeddings based on ordinal knowledge constraints. In Proceedings of ACL, pages 1501–1511, Beijing. find
  94. Liu, Xueqing, Yangqiu Song, Shixia Liu, and Haixun Wang. 2012. Automatic taxonomy construction from keywords. In Proceedings of KDD, pages 1433–1441, Beijing. find
  95. Luong, Thang, Richard Socher, and Christopher Manning. 2013. Better word representations with recursive neural networks for morphology. In Proceedings of CoNLL, pages 104–113, Sofia. our card
  96. Markman, Arthur B. and Edward J. Wisniewski. 1997. Similar and different: The differentiation of basic-level categories. find
  97. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23(1). McRae, Ken, George S. Cree, Mark S. Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37(4):547–559. find
  98. Medin, Douglas L., Mark W. Altom, and Timothy D. Murphy. 1984. Given versus induced category representations: Use of prototype and exemplar information in classification. Journal of Experimental Psychology, 10(3):333–352. find
  99. Mikolov, Tomas, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of ICLR: Workshop Papers, Scottsdale, AZ. http://arxiv.org/abs/1301.3781 ↗ our card
  100. Mikolov, Tomas, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of the 27th Annual Conference on Advances in Neural Information Processing Systems (NIPS), pages 3111–3119, South Lake Ta… our card
  101. Miller, George A. 1995. WordNet: A Lexical Database for English. Communications of the ACM, 38(11):39–41. our card
  102. Mohler, Michael, David Bracewell, Marc Tomlinson, and David Hinote. 2013. Semantic signatures for example-based linguistic metaphor detection. In Proceedings of the First Workshop on Metaphor in NLP, pages 27–35, Atlanta, GA. Mrkšić, Nikola, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina… find
  103. Murphy, Lynne M. 2003. Semantic Relations and the Lexicon: Antonymy, Synonymy and Other Paradigms. find
  104. Cambridge University Press. Myers, Raymond H. 1990. Classical and Modern Regression with Applications (Volume 2). Duxbury Press. find
  105. Neelakantan, Arvind, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient non-parametric estimation of multiple embeddings per word in vector space. In Proceedings of EMNLP, pages 1059–1069, Doha. our card
  106. Nelson, Douglas L., Cathy L. McEvoy, and Thomas A. Schreiber. 2004. The University of South Florida free association, rhyme, and word fragment norms. Behavior Research Methods, 36(3):402–407. Nivre, Joakim, Željko Agić, Maria Jesus Aranzabe, et al. 2015. Universal Dependencies 1.2. LINDAT/CLARIN di… find
  107. Ono, Masataka, Makoto Miwa, and Yutaka Sasaki. 2015. Word embedding-based antonym detection using thesauri and distributional information. In Proceedings of NAACL-HLT, pages 984–989, Denver, CO. find
  108. Osherson, Daniel N. and Edward E. Smith. 1981. On the adequacy of prototype theory as a theory of concepts. Cognition, 9(1):35–58. find
  109. Osherson, Daniel N. and Edward E. Smith. 1997. On typicality and vagueness. Cognition, 64(2):189–206. find
  110. Padó, Sebastian, Ulrike Padó, and Katrin Erk. 2007. Flexible, corpus-based modelling of human plausibility judgements. In Proceedings of EMNLP-CoNLL, pages 400–409, Prague. find
  111. Paivio, Allan. 1991. Dual coding theory: Retrospect and current status. Canadian Journal of Psychology, 45(3):255. find
  112. Pantel, Patrick and Marco Pennacchiotti. 2006. Espresso: Leveraging generic patterns for automatically harvesting semantic relations. In Proceedings of ACL, pages 113–120, Sydney. our card
  113. Parikh, Ankur, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of EMNLP, pages 2249–2255, Austin, TX. our card
  114. Pedersen, Ted, Siddharth Patwardhan, and Jason Michelizzi. 2004. WordNet::Similarity - Measuring the relatedness of concepts. In Proceedings of AAAI, pages 1024–1025, San Jose, CA. find
  115. Pereira, Jose Costa, Emanuele Coviello, Gabriel Doyle, Nikhil Rasiwasia, Gert R. G. Lanckriet, Roger Levy, and Nuno Vasconcelos. 2014. On the role of correlation and abstraction in cross-modal multimedia retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3):521–535. find
  116. Pulman, Stephan Guy. 1983. Word Meaning and Belief . Routledge. find
  117. Quillian, Ross M. 1967. Word concepts: A theory and simulation of some basic semantic capabilities. Behavioral Science, 12(5):410–430. find
  118. Rei, Marek and Ted Briscoe. 2014. Looking for hyponyms in vector space. In Proceedings of CoNLL, pages 68–77, Baltimore, MD. find
  119. Reisinger, Joseph and Raymond J. Mooney. 2010a. A mixture model with sharing for lexical semantics. In Proceedings of EMNLP, pages 1173–1182, Cambridge, MA. find
  120. Reisinger, Joseph and Raymond J. Mooney. 2010b. Multi-prototype vector-space models of word meaning. In Proceedings of NAACL-HLT, pages 109–117, Los Angeles, CA. our card
  121. Resnik, Philip. 1995. Using information content to evaluate semantic similarity in a taxonomy. In Proceedings of IJCAI, pages 448–453, Montreal. our card
  122. Riedel, Sebastian, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of NAACL-HLT, pages 74–84, Atlanta, GA. find
  123. Rimell, Laura. 2014. Distributional lexical entailment by topic coherence. In Proceedings of EACL, pages 511–519, Gothenburg. find
  124. Roller, Stephen and Katrin Erk. 2016. Relations such as hypernymy: Identifying and exploiting Hearst patterns in distributional vectors for lexical entailment. In Proceedings of EMNLP, pages 2163–2172, Austin, TX. find
  125. Roller, Stephen, Katrin Erk, and Gemma Boleda. 2014. Inclusive yet selective: Supervised distributional hypernymy detection. In Proceedings of COLING, pages 1025–1036, Dublin. our card
  126. Rosch, Eleanor H. 1973. Natural categories. Cognitive Psychology, 4(3):328–350. find
  127. Rosch, Eleanor H. 1975. Cognitive representations of semantic categories. Journal of Experimental Psychology, 104(3):192–233. find
  128. Rubinstein, Dana, Effi Levi, Roy Schwartz, and Ari Rappoport. 2015. How well do distributional models capture different types of semantic knowledge? In Proceedings of ACL, pages 726–730, Beijing. find
  129. Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. 2015. find
  130. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211–252. Sacaleanu, Bogdan, Constantin Orasan, Christian Spurk, Shiyan Ou, Oscar Ferrandez, Milen Kouylekov, and Matteo Negri. 2008. Entailment-based question answering for structured data. In Proceed… our card
  131. Santus, Enrico, Frances Yung, Alessandro Lenci, and Chu-Ren Huang. 2015. EVALution 1.0: An evolving semantic dataset for training and evaluation of distributional semantic models. In Proceedings of the 4th Workshop on Linked Data in Linguistics: Resources and Applications, pages 64–69, Beijing. find
  132. Schnabel, Tobias, Igor Labutov, David M. Mimno, and Thorsten Joachims. 2015. Evaluation methods for unsupervised word embeddings. In Proceedings of EMNLP, pages 298–307, Lisbon. our card
  133. Schwartz, Roy, Roi Reichart, and Ari Rappoport. 2015. Symmetric pattern based word embeddings for improved word similarity prediction. In Proceedings of CoNLL, pages 258–267, Beijing. find
  134. Shwartz, Vered, Yoav Goldberg, and Ido Dagan. 2016. Improving hypernymy detection with an integrated path-based and distributional method. In Proceedings of ACL, pages 2389–2398, Berlin. our card
  135. Shwartz, Vered, Omer Levy, Ido Dagan, and Jacob Goldberger. 2015. Learning to exploit structured resources for lexical inference. In Proceedings of CoNLL, pages 175–184, Beijing. find
  136. Shwartz, Vered, Enrico Santus, and Dominik Schlechtweg. 2017. Hypernyms under siege: Linguistically-motivated artillery for hypernymy detection. In Proceedings of EACL, pages 65–75, Valencia. find
  137. Silberer, Carina and Mirella Lapata. 2012. Grounded models of semantic representation. In Proceedings of EMNLP, pages 1423–1433, Jeju. find
  138. Silberer, Carina and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In Proceedings of ACL, pages 721–732, Baltimore, MD. our card
  139. Simonyan, Karen and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of ICLR, San Diego, CA. http://arxiv.org/abs/1409.1556 ↗ our card
  140. Snow, Rion, Daniel Jurafsky, and Andrew Y. Ng. 2004. Learning syntactic patterns for automatic hypernym discovery. In Proceedings of NIPS, pages 1297–1304, Vancouver. find
  141. Snow, Rion, Daniel Jurafsky, and Andrew Y. Ng. 2006. Semantic taxonomy induction from heterogenous evidence. In Proceedings of ACL, pages 801–808, Sydney. find
  142. Socher, Richard, Danqi Chen, Christopher D. Manning, and Andrew Y. Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In Proceedings of NIPS, pages 926–934, South Lake Tahoe, NV. our card
  143. Socher, Richard, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. Transactions of the ACL, 2:207–218, Baltimore, MD. our card
  144. Song, Yangqiu, Haixun Wang, Zhongyuan Wang, Hongsong Li, and Weizhu Chen. 2011. Short text conceptualization using a probabilistic knowledgebase. In Proceedings of AAAI, pages 2330–2336, San Francisco, CA. find
  145. Suchanek, Fabian M., Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: A core of semantic knowledge. In Proceedings of WWW, pages 697–706, Banff. Tanon, Thomas Pellissier, Denny Vrandečić, Sebastian Schaffert, Thomas Steiner, and Lydia Pintscher. 2016. From Freebase to Wikidata: The great migration… find
  146. Linguistic Categorization. Oxford University Press. Tsvetkov, Yulia, Manaal Faruqui, Wang Ling, Guillaume Lample, and Chris Dyer. 2015. Evaluation of word vector representations by subspace alignment. In Proceedings of EMNLP, pages 2049–2054, Lisbon. find
  147. Turney, Peter D. 2006. Similarity of semantic relations. Computational Linguistics, 32(3):379–416. our card
  148. Turney, Peter D. and Saif M. Mohammad. 2015. Experiments with three approaches to recognizing lexical entailment. Natural Language Engineering, 21(3):437–476. find
  149. Turney, Peter D. and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. our card
  150. Vendrov, Ivan, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2016. Order-embeddings of images and language. In Proceedings of ICLR, San Juan, PR. find
  151. Vilnis, Luke and Andrew McCallum. 2015. Word representations via Gaussian embedding. In Proceedings of ICLR, San Diego, CA. find
  152. Šuster, Simon, Ivan Titov, and Gertjan van Noord. 2016. Bilingual learning of multi-sense embeddings with discrete autoencoders. In Proceedings of NAACL-HLT, pages 1346–1356, San Diego, CA. Vulić, Ivan and Anna Korhonen. 2016. Is “universal syntax” universally useful for learning distributed word r… find
  153. Vylomova, Ekaterina, Laura Rimell, Trevor Cohn, and Timothy Baldwin. 2016. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning. In Proceedings of ACL, pages 1671–1682, Berlin. find
  154. Weeds, Julie, Daoud Clarke, Jeremy Reffin, David Weir, and Bill Keller. 2014. Learning to distinguish hypernyms and co-hyponyms. In Proceedings of COLING, pages 2249–2259, Dublin. our card
  155. Weeds, Julie and David Weir. 2003. A general framework for distributional similarity. In Proceedings of EMNLP, pages 81–88, Sapporo. our card
  156. Weeds, Julie, David Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In Proceedings of COLING, pages 1015–1021, Geneva. find
  157. Wieting, John, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. From paraphrase database to compositional paraphrase model and back. Transactions of the ACL, 3:345–358. our card
  158. Wu, Wentao, Hongsong Li, Haixun Wang, and Kenny Q. Zhu. 2012. Probase: A probabilistic taxonomy for text understanding. In Proceedings of SIGMOD, pages 481–492, Scottsdale, AZ. find
  159. Wu, Zhibiao and Martha Palmer. 1994. Verb semantics and lexical selection. In Proceedings of ACL, pages 133–138, Las Cruces, NM. our card
  160. Yaghoobzadeh, Yadollah and Hinrich Schütze. 2016. Intrinsic subspace evaluation of word embedding representations. In Proceedings of ACL, pages 236–246, Berlin. find
  161. Yih, Scott Wen-Tau, Geoffrey Zweig, and John C. Platt. 2012. Polarity inducing latent semantic analysis. In Proceedings of EMNLP, pages 1212–1222, Jeju. find
  162. Young, Peter, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the ACL, 2:67–78. find
  163. Yu, Mo and Mark Dredze. 2014. Improving lexical embeddings with semantic knowledge. In Proceedings of ACL, pages 545–550, Baltimore, MD. our card
  164. Zhila, Alisa, Wen-tau Yih, Christopher Meek, Geoffrey Zweig, and Tomas Mikolov. 2013. Combining heterogeneous models for measuring relational similarity. In Proceedings of NAACL-HLT, pages 1000–1009, Atlanta, GA. find
  165. Zhitomirsky-Geffet, Maayan and Ido Dagan. 2009. Bootstrapping distributional feature vector quality. Computational Linguistics, 35(3):435–461. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vulic-2017-hyperlex-large-scale-evaluation-graded/