Elamur Research
☰
home / papers / 2015 / Do Supervised Distributional Methods Really Learn Lexical Inference Relations?

Do Supervised Distributional Methods Really Learn Lexical Inference Relations?

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/N15-1098.pdf

Summary

This paper asks whether supervised classifiers built on distributional word vectors actually learn lexical inference relations, such as hypernymy and entailment, between two words x and y. The authors built 9 word representations from Wikipedia (1.5 billion tokens) by combining three context types (bag-of-words, positional, dependency) with three models (PPMI, SVD, SGNS). They tested the pair compositions concat, diff, only x and only y using logistic regression and SVM classifiers on five labeled datasets. They found that much of the reported success comes from lexical memorization. When train and test vocabularies are kept apart, the classifiers mostly judge whether y looks like a typical category word, a "prototypical hypernym", and largely ignore x. A kernel analysis explains why: linear and polynomial kernels over concat or diff have no term for the similarity between x and y. Adding that similarity through new kernels helps only slightly. The authors conclude that contextual features of individual words may not contain relational information at all, which matters for anyone designing entailment or hypernymy classifiers.

Key points

  • Datasets: Kotlerman 2010, BLESS 2011, Baroni 2012, Turney 2014 and Levy 2014. The authors released splits with zero lexical overlap between train and test for replication.
  • Lexical memorization: A classifier using only a one-hot encoding of y comes within 10 F1 points of contextual-feature classifiers on the larger datasets.
  • Weak baselines beat or match supervision:
    • Unsupervised cosine similarity outperforms every supervised method on 2 of 5 datasets.
    • The only-y composition is within 5 F1 points of the best configuration on 4 of 5 datasets.
  • Mismatched pairs test: On synthetic pairs like (banana, animal), the false-positive rate tracks recall almost exactly (match error ≈ 0.935 · recall). Top-weighted features include Hearst-pattern cues such as "such", "other" and "including".
  • Kernel fixes: An RBF kernel over diff and a new SIM kernel that includes x·y reduce match error to ≈ 0.618 · recall. However, F1 improves only marginally.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

118 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 115 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/n15-1098

@inproceedings{levy2015do,
  title = {{Do Supervised Distributional Methods Really Learn Lexical Inference Relations?}},
  author = {Levy, Omer and Remus, Steffen and Biemann, Chris and Dagan, Ido},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2015},
  doi = {10.3115/v1/n15-1098},
  url = {https://doi.org/10.3115/v1/n15-1098}
}

References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673– 721. find our card
  2. Marco Baroni and Alessandro Lenci. 2011. How we blessed distributional semantic evaluation. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pages 1–10, Edinburgh, UK. find
  3. Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012. Entailment above the word level in distributional semantics. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 23–32, Avignon, France. find
  4. Chris Biemann and Martin Riedl. 2013. Text: Now in 2D! A framework for lexical expansion with contextual similarity. Journal of Language Modelling, 1(1):55–95. find
  5. John Caron. 2001. Experiments with LSA scoring: Optimal rank and basis. In Proceedings of the SIAM Computational Information Retrieval Workshop, pages 157–169. find
  6. Daoud Clarke. 2009. Context-theoretic semantics for natural language: an overview. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 112–119, Athens, Greece. find
  7. Ruiji Fu, Jiang Guo, Bing Qin, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014. Learning semantic hierarchies via word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1199–1209, Baltimore, Maryland. find
  8. Yoav Goldberg and Jon Orwant. 2013. A dataset of syntactic-ngrams over time from a very large corpus of english books. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 241–24… find
  9. Marti A Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics, pages 529–545, Nantes, France. find
  10. David A Jurgens, Peter D Turney, Saif M Mohammad, and Keith J Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume… find
  11. Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geffet. 2010. Directional distributional similarity for lexical inference. Natural Language Engineering, 4(16):359–389. find
  12. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302–308, Baltimore, Maryland. find
  13. Omer Levy, Ido Dagan, and Jacob Goldberger. 2014. Focused entailment graphs for open ie propositions. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 87–97, Baltimore, Maryland. find
  14. Dekang Lin. 1998. Automatic retrieval and clustering of similar words. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2, pages 768–774, Montréal, Quebec, Canada. find
  15. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of the International Conference on Learning Representations (ICLR). find
  16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  17. Yoshiki Niwa and Yoshihiko Nitta. 1994. Co-occurrence vectors from corpora vs. distance vectors from dictionaries. In COLING 1994 Volume 1: The 15th International Conference on Computational Linguistics, Kyoto, Japan. find
  18. Sebastian Padó and Mirella Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199. find our card
  19. Laura Rimell. 2014. Distributional lexical entailment by topic coherence. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 511–519, Gothenburg, Sweden. find
  20. Stephen Roller, Katrin Erk, and Gemma Boleda. 2014. Inclusive yet selective: Supervised distributional hypernymy detection. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1025– 1036, Dublin, Ireland. find
  21. Enrico Santus, Alessandro Lenci, Qin Lu, and Sabine Schulte im Walde. 2014. Chasing hypernyms in vector spaces with entropy. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 38–42, Gothenburg, Sweden. find
  22. Hinrich Schütze. 1993. Part-of-speech induction from scratch. In Proceedings of the 31st Annual Meeting of the Association for Computational Linguistics, pages 251–258, Columbus, Ohio, USA. find
  23. Rion Snow, Daniel Jurafsky, and Andrew Y Ng. 2005. Learning syntactic patterns for automatic hypernym discovery. In Advances in Neural Information Processing. find
  24. Peter D Turney and Saif M Mohammad. 2014. Experiments with three approaches to recognizing lexical entailment. Natural Language Engineering, pages 1–40. find
  25. Peter D Turney. 2006. Similarity of semantic relations. Computational Linguistics, 32(3):379–416. find
  26. Julie Weeds and David Weir. 2003. A general framework for distributional similarity. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 81–88, Sapporo, Japan. find
  27. Julie Weeds, Daoud Clarke, Jeremy Reffin, David Weir, and Bill Keller. 2014. Learning to distinguish hypernyms and co-hyponyms. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2249– 2259, Dublin, Ireland. find