Elamur Research⌕
☰
home / papers / 2015 / Deriving Boolean structures from distributional vectors

Deriving Boolean structures from distributional vectors

Paper · 2015 · German Kruszewski, Denis Paperno and Marco Baroni

Summary

Distributional semantic models scale to huge vocabularies and capture how related words are, but they handle logical relations such as entailment poorly. Formal, model-theoretic semantics treats those relations elegantly but cannot be learned from data at scale. Kruszewski, Paperno and Baroni combine the two by learning a mapping from distributional vectors into Boolean-like vectors, in which entailment becomes feature inclusion: the active bits of the more specific expression must be contained in those of the more general one. The inputs are word vectors and sentence vectors composed with the method of Paperno et al. (2014). The resulting Boolean Distributional Semantic Model (BDSM) is trained on entailing and non-entailing pairs. Only the mapping weights are learned, while the inclusion-based inference step is fixed in advance. On lexical and sentential entailment, BDSM matches a strong SVM baseline, beats it on sentences, and degrades less when training data shrink. The induced vectors are compact and partly interpretable. They shift the space away from plain relatedness toward overlap of denotations, a promising route to logic-aware distributional semantics.

Key points

  • Model: a sigmoid with a "temperature" parameter is applied to a linear transform (W, b), relaxed to the interval [0, 1]^H, and trained with MSE on entailment predictions. The main baseline, an SVM trained directly on distributional vectors, builds on earlier work by Weeds et al. (2014), Roller et al. (2014) and Baroni et al. (2012).
  • Interpretability and efficiency: each word gets one compact Boolean vector that alone suffices to check its entailments, which can reduce memory and compute needs. The number of active dimensions correlates with how general or abstract a word is.
  • Qualitative test set: 624 related word pairs, built from the 10 nearest neighbours of the 10K most frequent words and filtered to the 60K–90K frequency range of a 2.8B-token corpus. Cosines in the two spaces were z-normalized before comparison.
  • Findings on that set: co-hyponyms (137 pairs) and personal names (97 pairs) become less similar after mapping (p < 0.001). Synonyms (49) and derivationally related words (10) become more similar (p < 0.01). The extensional reading holds only as a tendency; for example, man and woman still overlap considerably.
  • Publication: TACL vol. 3 (2015), pp. 375–388, released under CC-BY-NC-SA 4.0 and supported by the ERC COMPOSES grant. Planned future work covers negation, conjunction and enforcing disjointness between co-hyponyms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TrentoTrento · Italy85 works · 15,524 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as German Kruszewski, Denis Paperno and Marco Baroni. 2015. Deriving Boolean structures from distributional vectors. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00145

@article{kruszewski2015deriving,
  title = {{Deriving Boolean structures from distributional vectors}},
  author = {German Kruszewski, Denis Paperno and Marco Baroni},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.1162/tacl\_a\_00145},
  url = {https://doi.org/10.1162/tacl_a_00145}
}

References

45 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni and Alessandro Lenci. 2010. Distributional Memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673–721.
  2. Marco Baroni and Alessandro Lenci. 2011. How we BLESSed distributional semantic evaluation. In Proceedings of the EMNLP GEMS Workshop, pages 1–10, Edinburgh, UK.
  3. Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-Chieh Shan. 2012. Entailment above the word level in distributional semantics. In Proceedings of EACL, pages 23–32, Avignon, France.
  4. Marco Baroni, Raffaella Bernardi, and Roberto Zamparelli. 2014a. Frege in space: A program for compositional distributional semantics. Linguistic Issues in Language Technology, 9(6):5–110. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014b. Don’t count, predict! a systematic comparison of c…
  5. Islam Beltagy, Cuong Chau, Gemma Boleda, Dan Garrette, Katrin Erk, and Raymond Mooney. 2013. Montague meets Markov: Deep semantics with probabilistic logical form. In Proceedings of *SEM, pages 11– 21, Atlanta, GA. find
  6. William Blacoe and Mirella Lapata. 2012. A comparison of vector-based representations for semantic composition. In Proceedings of EMNLP, pages 546–556, Jeju Island, Korea.
  7. Antoine Bordes, Xavier Glorot, Jason Weston, and Yoshua Bengio. 2012. Joint learning of words and meaning representations for open-text semantic parsing. In Proceedings of AISTATS, pages 127–135, La Palma, Canary Islands. find
  8. Samuel R. Bowman. 2013. Can recursive neural tensor networks learn logical reasoning? CoRR, abs/1312.6192. find
  9. Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2013. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research methods, pages 1–8. find
  10. Gennaro Chierchia and Sally McConnell-Ginet. 2000. Meaning and Grammar: An Introduction to Semantics. find
  11. MIT Press, Cambridge, MA. Kenneth Church and Peter Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22–29.
  12. Stephen Clark and Stephen Pulman. 2007. Combining symbolic and distributional models of meaning. In Proceedings of the First Symposium on Quantum Interaction, pages 52–55, Stanford, CA. find
  13. Daoud Clarke. 2012. A context-theoretic framework for compositionality in distributional semantics. Computational Linguistics, 38(1):41–71.
  14. Bob Coecke, Mehrnoosh Sadrzadeh, and Stephen Clark. 2010. Mathematical foundations for a compositional distributional model of meaning. Linguistic Analysis, 36:345–384. find
  15. Nello Cristianini and John Shawe-Taylor. 2000. An introduction to Support Vector Machines and other kernel-based learning methods. find
  16. Cambridge University Press, Cambridge, UK. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2009. Recognizing textual entailment: rationale, evaluation and approaches. Natural Language Engineering, 15:459–476. find
  17. Katrin Erk. 2009. Supporting inferences in semantic space: Representing words as regions. In Proceedings of IWCS, pages 104–115, Tilburg, Netherlands. find
  18. Katrin Erk. 2013. Towards a semantics for distributional representations. In Proceedings of IWCS, pages 95–106, Potsdam, Germany. Association for Computational Linguistics. Christiane Fellbaum, editor. 1998. find
  19. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA. Dan Garrette, Katrin Erk, and Ray Mooney. 2013. A formal approach to linking logical form and vector-space lexical semantics. In H. Bunt, J. Bos, and S. Pulman, editors, Computing Meaning, Vol. 4, pages 27– 48. find
  20. Springer, Berlin. Maayan Geffet and Ido Dagan. 2005. The distributional inclusion hypotheses and lexical entailment. In Proceedings of ACL, pages 107–114, Ann Arbor, MI.
  21. Edward Grefenstette. 2013. Towards a formal distributional semantics: Simulating logical calculi with tensors. Proceedings of *SEM, pages 1–10. find
  22. Rodolphe Jenatton, Nicolas Le Roux, Antoine Bordes, and Guillaume Obozinski. 2012. A latent factor model for highly multi-relational data. In Proceedings of NIPS, pages 3176–3184, La Palma, Canary Islands. find
  23. Douwe Kiela, Felix Hill, Anna Korhonen, and Stephen Clark. 2014. Improving multi-modal representations using image dispersion: Why less is sometimes more. In Proceedings of ACL, pages 835–841, Baltimore, MD. find
  24. Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geffet. 2010. Directional distributional similarity for lexical inference. Natural Language Engineering, 16(4):359–389. find
  25. Thomas Landauer and Susan Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211–240. find
  26. Alessandro Lenci and Giulia Benotto. 2012. Identifying hypernyms in distributional semantic spaces. In Proceedings of *SEM, pages 75–79, Montreal, Canada.
  27. Mike Lewis and Mark Steedman. 2013. Combined distributional and logical semantics. Transactions of the Association for Computational Linguistics, 1:179– 192. find
  28. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014a. SemEval-2014 Task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of SemEval, pages 1–…
  29. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. http://arxiv.org/abs/ 1301.3781/. ↗
  30. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  31. Tom M Mitchell. 1980. The need for biases in learning generalizations. Technical report, New Brunswick, NJ. find
  32. Denis Paperno, Nghia The Pham, and Marco Baroni. 2014. A practical and linguistically-motivated approach to compositional distributional semantics. In Proceedings of ACL, pages 90–99, Baltimore, MD. find
  33. Tamara Polajnar, Luana Fagarasan, and Stephen Clark. 2014. Reducing dimensions of tensors in type-driven distributional semantics. In Proceedings of EMNLP, pages 1036–1046, Doha, Qatar. find
  34. Laura Rimell. 2014. Distributional lexical entailment by topic coherence. In Proceedings of EACL, pages 511– 519, Gothenburg, Sweden. Tim Rocktäschel, Sameer Singh, and Sebastian Riedel. 2015. Injecting logical background knowledge into embeddings for relation extraction. In Proceedings of HLT-NAAC… find
  35. Stephen Roller, Katrin Erk, and Gemma Boleda. 2014. Inclusive yet selective: Supervised distributional hypernymy detection. In Proceedings of COLING, pages 1025–1036, Dublin, Ireland. Enrico Santus, Alessandro Lenci, Qin Lu, and Sabine Schulte im Walde. 2014. Chasing hypernyms in vector spaces with…
  36. T. Sasao. 1999. Switching Theory for Logic Synthesis. Springer, London. Hinrich Schütze. 1997. Ambiguity Resolution in Natural Language Learning. find
  37. CSLI, Stanford, CA. Richard Socher, Danqi Chen, Christopher D. Manning, and Andrew Y. Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In Proceedings of NIPS, pages 926–934, Lake Tahoe, NV.
  38. Peter Turney and Said Mohammad. 2014. Experiments with three approaches to recognizing lexical entailment. Natural Language Engineering. In press. find
  39. Peter Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188.
  40. Peter Turney, Yair Neuman, Dan Assaf, and Yohai Cohen. 2011. Literal and metaphorical sense identification through concrete and abstract context. In Proceedings of EMNLP, pages 680–690, Edinburgh, UK.
  41. Peter Turney. 2012. Domain and function: A dual-space model of semantic relations and compositions. Journal of Artificial Intelligence Research, 44:533–585. find
  42. Eva Maria Vecchi, Marco Baroni, and Roberto Zamparelli. 2011. (Linear) maps of the impossible: Capturing semantic anomalies in distributional space. In Proceedings of the ACL Workshop on Distributional Semantics and Compositionality, pages 1–9, Portland, OR. find
  43. Julie Weeds, David Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In Proceedings of COLING, pages 1015–1021, Geneva, Switzerland. find
  44. Julie Weeds, Daoud Clarke, Jeremy Reffin, David Weir, and Bill Keller. 2014. Learning to distinguish hypernyms and co-hyponyms. In Proceedings of COLING, pages 2249–2259, Dublin, Ireland.
  45. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kruszewski-2015-deriving-boolean-structures-distributional-vectors/