Elamur Research⌕
☰
home / papers / 2012 / Entailment above the word level in distributional semantics

Entailment above the word level in distributional semantics

Summary

Distributional semantics (DS) has been strong at measuring word similarity, while formal semantics (FS) handles inference and quantification. This paper asks whether phrase vectors taken directly from a corpus carry enough information to detect entailment beyond single words. It runs two experiments.

In the first, the authors use the fact that an adjective-noun phrase almost always entails its head noun (big cat entails cat). These AN–N vector pairs become free training data for an SVM classifier, which is then applied to noun–noun lexical entailment (dog entails animal). With almost no manual labelling, it performs about as well as the hand-designed balAPinc measure.

In the second, the authors train an SVM on pairs of quantifier-noun phrases with the same noun (many dogs entails some dogs). The classifier generalizes to quantifiers it never saw during training. The AN-trained classifier and balAPinc both fail on this task, which suggests that quantifier-phrase entailment has different distributional signals than noun entailment. FS type theory predicts exactly this split. The work links DS with core FS concerns, including function words, and points toward textual entailment applications.

Key points

  • Experiment 1: an SVM trained only on automatically harvested AN |= N pairs detects lexical entailment between nouns. Its performance nearly matches balAPinc (Kotlerman et al., 2010), a feature-inclusion measure.

  • Experiment 2, evaluation regimes: QN entailment is tested under two regimes:

    • SVM pair-out, which holds out quantifier pairs;
    • SVM quantifier-out, which holds out whole quantifiers.

    Quantifier-out matches the lexical classifiers from Experiment 1, and pair-out does even better.

  • Quantifier-out breakdown (Table 4):

    • The data has 15,074 entailing and 16,910 non-entailing instances, with 71% overall accuracy.
    • Universal-like quantifiers score high (each 98%, every 95%).
    • Existential-like ones score low (both 44%, either 34%).

  • Methods that fail on QNs: SVM AN |= N and balAPinc do no better than the baselines. Their tuned balAPinc thresholds also diverge sharply (t = 0.01 vs. 0.26).

  • Quantifier-only vectors: using the quantifier vectors alone, without the noun, still beats chance. However, performance drops sharply in the held-out-quantifier setting, which points to memorized pair-specific patterns.

Interesting ideas

  • I1. Free supervision from a structural regularity. A construction that almost always entails one of its parts (AN entails N) can supply labelled training pairs without annotation. The same trick could bootstrap relation classifiers for low-resource languages such as Ukrainian.
  • I2. Holding out whole categories, not just items. Leaving out an entire quantifier tests real generalization rather than memorization. The paper's quantifier-only model shows how large the gap between these regimes can be.
  • I3. Watching for easy negatives. Negatives made by randomly swapping nouns roughly halved error rates, so the authors excluded them from the reported results. This is a useful reminder that careless negative sampling can inflate benchmark scores.
  • I4. Using cross-task transfer as a diagnostic. When a model trained on one entailment type fails on another, that failure is evidence that the two relations differ in kind. This is a cheap way to probe whether two phenomena share an underlying signal.
  • I5. Per-class error breakdowns that reveal semantic groupings. Splitting results by quantifier exposed a universal-versus-existential divide. Fine-grained breakdowns like this can surface linguistically meaningful patterns that a single overall score would hide.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

112 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 108 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{baroni2012entailment,
  title = {{Entailment above the word level in distributional semantics}},
  author = {Baroni, Marco and Bernardi, Raffaella and Do, Ngoc-Quynh and Shan, Chung-chieh},
  booktitle = {Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2012},
  url = {https://aclanthology.org/E12-1004/}
}

References

37 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Timothy Baldwin, Colin Bannard, Takaaki Tanaka, and Dominic Widdows. 2003. An empirical model of multiword expression decomposability. In Proceedings of the ACL 2003 Workshop on Multiword Expressions, pages 89–96. find our card
  2. Marco Baroni and Alessandro Lenci. 2011. How we BLESSed distributional semantic evaluation. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics. find our card
  3. Marco Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of EMNLP, pages 1183–1193, Boston, MA. find
  4. Johan Bos and Katja Markert. 2006. When logical inference helps determining textual entailment (and when it doesn’t. In Proceedings of the Second PAS-CAL Challenges Workshop on Recognising Textual Entailment. find
  5. Paul Buitelaar and Philipp Cimiano. 2008. Bridging the Gap between Text and Knowledge. IOS, Amsterdam. find
  6. Nathanael Chambers, Daniel Cer, Trond Grenager, David Hall, Chloe Kiddon, Bill MacCartney, Marie-Catherine de Marneffe, Daniel Ramage, Eric Yeh, and Christopher D. Manning. 2007. Learning alignments and leveraging natural logic. In ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. find
  7. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):27:1–27:27. find
  8. Kenneth Church and Peter Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22–29. find our card
  9. Nello Cristianini and John Shawe-Taylor. 2000. An introduction to Support Vector Machines and other kernel-based learning methods. Cambridge University Press, Cambridge. find
  10. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2009. Recognizing textual entailment: rational, evaluation and approaches. Natural Language Engineering, 15:459–476. find
  11. Katrin Erk. 2009. Supporting inferences in semantic space: representing words as regions. In Proceedings of IWCS, pages 104–115, Tilburg, Netherlands. find
  12. Maayan Geffet and Ido Dagan. 2005. The distributional inclusion hypotheses and lexical entailment. In Proceedings of ACL, pages 107–114, Ann Arbor, MI. find our card
  13. Edward Grefenstette and Mehrnoosh Sadrzadeh. 2011. Experimental support for a categorical compositional distributional model of meaning. In Proceedings of EMNLP, pages 1395–1404, Edinburgh. find
  14. Emiliano Guevara. 2010. A regression model of adjective-noun compositionality in distributional semantics. In Proceedings of the ACL GEMS Workshop, pages 33–37, Uppsala, Sweden. find
  15. Marti Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Proceedings of COLING, pages 539–545, Nantes, France. find
  16. Irene Heim and Angelika Kratzer. 1998. Semantics in Generative Grammar. Blackwell, Oxford. find
  17. Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geffet. 2010. Directional distributional similarity for lexical inference. Natural Language Engineering, 16(4):359–389. find
  18. Milen Kouleykov and Bernardo Magnini. 2005. Tree edit sistance for textual entailment. In Proceedings of RALNP-2005, International Conference on Recent Advances in Natural Language Processing, pages 271–278. find
  19. Thomas Landauer and Susan Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211–240. find
  20. Dekang Lin. 1998. An information-theoretic definition of similarity. In Proceedings of ICML, pages 296–304, Madison, WI, USA. find
  21. Kevin Lund and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, 28:203– 208. find
  22. Chris Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press, Cambridge. find
  23. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  24. Richard Montague. 1970. Universal Grammar. Theoria, 36:373–398. find
  25. Patrick Pantel and Deepak Ravichandran. 2004. Automatically labeliing semantic classes. In Proceedings of HLT-NAACL 2004, pages 321–328. find
  26. Reinhard Rapp. 2003. Word sense discovery based on sense descriptor dissimilarity. In Proceedings of the 9th MT Summit, pages 315–322, New Orleans, LA. find
  27. Magnus Sahlgren. 2006. The Word-Space Model. Dissertation, Stockholm University. find
  28. Helmut Schmid. 1995. Improvements in part-of-speech tagging with an application to German. In Proceedings of the EACL-SIGDAT Workshop, Dublin, Ireland. find
  29. Hinrich Schütze. 1997. Ambiguity Resolution in Natural Language Learning. CSLI, Stanford, CA. find
  30. Rion Snow, Daniel Juravsky, and Andrew Y. Ng. 2005. Learning syntactic patterns for automatic hypernym discovery. In Proceedings of NIPS 17. find
  31. Rion Snow, Daniel Juravsky, and Andrew Y. Ng. 2006. Semantic taxonomy induction from heterogenous evidence. In Proceedings of ACL 2006, pages 801–808. find
  32. Richmond H. Thomason, editor. 1974. Formal Philosophy: Selected Papers of Richard Montague. Yale University Press, New York. find
  33. Peter Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188. find
  34. Peter Turney. 2008. A uniform approach to analogies, synonyms, antonyms and associations. In Proceedings of COLING, pages 905–912, Manchester, UK. find
  35. Julie Weeds, David Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In Proceedings of the 20th International Conference of Computational Linguistics, COLING-2004, pages 1015–1021. find
  36. Fabio M. Zanzotto, Marco Pennacchiotti, and Alessandro Moschitti. 2007. Shallow semantics in fast textual entailment rule learners. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. find
  37. Maayan Zhitomirsky-Geffet and Ido Dagan. 2010. Bootstrapping distributional feature vector quality. Computational Linguistics, 35(3):435–461. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baroni-2012-entailment-above-word-level-distributional/