Elamur Research⌕
☰
home / papers / 2012 / A Context-Theoretic Framework for Compositionality in Distributional Semantics

A Context-Theoretic Framework for Compositionality in Distributional Semantics

Summary

Distributional methods capture word meaning well, but there is no agreed theory of how word vectors should combine into representations of phrases and sentences. It is also unclear how such representations relate to logic-based semantics. Corpus counts cannot solve this alone, because longer word strings are too rare. Daoud Clarke tackles the problem by formalizing the idea of meaning as context. In this model, a string is represented by a vector of the contexts it would appear in within an idealized, infinite corpus. Abstracting the properties of this model gives the context-theoretic framework:

  • meanings are elements of a real unital associative algebra;
  • composition is bilinear and associative;
  • entailment is a vector lattice ordering;
  • a graded degree of entailment is defined as a conditional probability linked to string probability.

The article shows that many published composition methods are special cases of this framework. It also reformulates several textual entailment techniques inside it. As a result, any compatible method automatically gains a principled entailment measure, and the framework links vector-based and logical approaches to meaning.

Key points

  • A context theory is formally a tuple of five parts: an alphabet, a unital algebra over the reals, a map from symbols into the algebra, an abstract Lebesgue space, and an injective linear map from the algebra into that space.
  • Composition methods shown to fit the framework include vector addition (Landauer and Dumais; Foltz, Kintsch, and Landauer), the tensor product (Smolensky; Clark and Pulman; Widdows), Mitchell and Lapata's (2008) multiplication-based models, and matrix multiplication (Baroni and Zamparelli; Rudolph and Giesbrecht). The more elaborate model of Clark, Coecke, and Sadrzadeh (2008) also fits.
  • Describing these methods in the framework exposes a lattice structure in their vector spaces that earlier work overlooked. Each method can then be equipped with an entailment relation.
  • Recognizing textual entailment approaches recast in the framework include subsequence matching, the lexical entailment probability of Glickman and Dagan, and a model based on latent Dirichlet allocation (LDA). The LDA model is commutative, which the author treats as a simplification.
  • Planned future work includes evaluation on the Recognising Textual Entailment Challenge datasets and learning finite-dimensional algebras from corpora (with the University of Sussex). Other directions are computational feasibility, links to free probability, and how the framework handles multi-word expressions and non-compositionality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Institutions

🎓University of HertfordshireHatfield · United Kingdom2 works · 318 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00084

@article{clarke2012context,
  title = {{A Context-Theoretic Framework for Compositionality in Distributional Semantics}},
  author = {Clarke, Daoud},
  journal = {Computational Linguistics},
  year = {2012},
  doi = {10.1162/coli\_a\_00084},
  url = {https://doi.org/10.1162/coli_a_00084}
}

References

50 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abramovich, Yuri A. and Charalambos D. Aliprantis. 2002. An Invitation to Operator Theory. American Mathematical Society, Providence, RI. find
  2. Bar-Haim, Roy, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, pages 1–9, Venice. find
  3. Baroni, Marco and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2010), pages 1183–1193, East Stroudsburg PA. find
  4. Bellegarda, Jerome R. 2000. Exploiting latent semantic information in statistical language modeling. Proceedings of the IEEE, 88(8):1279–1296. find
  5. Blackburn, Patrick and Johan Bos. 2005. Representation and Inference for Natural Language. CSLI Publications, Stanford, CA. find
  6. Blei, David M., Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022. find our card
  7. Boyd-Graber, Jordan, David Blei, and Xiaojin Zhu. 2007. A topic model for word sense disambiguation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 1024–1033, Prague. find
  8. Cai, Junfu, Wee Sun Lee, and Yee Whye Teh. 2007. Improving word sense disambiguation using topic features. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 1015–1023, Prague. find
  9. Choi, Freddy, Peter Wiemer-Hastings, and Johanna Moore. 2001. Latent Semantic Analysis for text segmentation. find
  10. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing, pages 109–117, Ithaca, NY. find
  11. Clark, Stephen, Bob Coecke, and Mehrnoosh Sadrzadeh. 2008. A compositional distributional model of meaning. In Proceedings of the Second Quantum Interaction Symposium (QI-2008), pages 133–140, Oxford. find
  12. Clark, Stephen and Stephen Pulman. 2007. Combining symbolic and distributional models of meaning. In Proceedings of the AAAI Spring Symposium on Quantum Interaction, pages 52–55, Stanford, CA. find
  13. Clarke, Daoud. 2006. Meaning as context and subsequence analysis for textual entailment. In Proceedings of the Second PASCAL Recognising Textual Entailment Challenge, pages 134–139, Venice. find
  14. Clarke, Daoud. 2007. Context-theoretic Semantics for Natural Language: An Algebraic Framework. Ph.D. thesis, Department of Informatics, University of Sussex. find
  15. Collins, Michael. 1997. Three generative, lexicalised models for statistical parsing. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics, pages 16–23, Madrid. find our card
  16. Dagan, Ido, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Proceedings of the PASCAL Challenges Workshop on Recognising Textual Entailment, pages 1–8, Southampton, UK. find
  17. Deerwester, Scott, Susan Dumais, George Furnas, Thomas Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391–407. find
  18. Erk, Katrin and Sebastian Padó. 2009. Paraphrase assessment in structured vector space: Exploring parameters and datasets. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 57–65, Athens. find
  19. Firth, John R. 1968. A synopsis of linguistic theory, 1930–1955. In John R. Firth, editor, Selected Papers of JR Firth, 1952–59. Indiana University Press, Bloomington, pages 168–205. find
  20. Foltz, Peter W., Walter Kintsch, and Thomas K. Landauer. 1998. The measurement of textual coherence with latent semantic analysis. Discourse Process, 15:285–307. find
  21. Geffet, M. and I. Dagan. 2005. The distributional inclusion hypotheses and lexical entailment. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 107–114, Ann Arbor, MI. find our card
  22. Glickman, O. and I. Dagan. 2005. A probabilistic setting and lexical cooccurrence model for textual entailment. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, pages 43–48, Ann Arbor, MI. find
  23. Grefenstette, Edward, Mehrnoosh Sadrzadeh, Stephen Clark, Bob Coecke, and Stephen Pulman. 2011. Concrete sentence spaces for compositional distributional models of meaning. Proceedings of the 9th International Conference on Computational Semantics (IWCS 2011), pages 125–134, Oxford. find
  24. Grefenstette, Gregory. 1994. Explorations in Automatic Thesaurus Discovery. Kluwer Academic Publishers, Dordrecht, Netherlands. find
  25. Guevara, Emiliano. 2011. Computing semantic compositionality in distributional semantics. In Proceedings of the 9th International Conference on Computational Semantics (IWCS 2011), pages 135–144, Oxford. find
  26. Halmos, Paul. 1974. Finite Dimensional Vector Spaces. Springer, Berlin. find
  27. Harris, Zellig. 1968. Mathematical Structures of Language. Wiley, New York. find
  28. Hofmann, Thomas. 1999. Probabilistic latent semantic analysis. In Proceedings of the 15th Conference on Uncertainty in AI, pages 289–296, Stockholm. find our card
  29. Kamp, Hans and Uwe Reyle. 1993. From Discourse to Logic: Introduction to Model-theoretic Semantics of Natural Language, Formal Logic and Discourse Representation Theory, volume 42 of Studies in Linguistics and Philosophy. Kluwer, Dordrecht. find
  30. Kintsch, Walter. 2001. Predication. Cognitive Science, 25:173–202. find
  31. Lambek, Joachim. 1958. The mathematics of sentence structure. American Mathematical Monthly, 65:154–169. find
  32. Lambek, Joachim. 1961. On the calculus of syntactic types. In Roman Jakobson, editor, Structure of Language and Its Mathematical Aspects, pages 166–178, American Mathematical Society, Providence, RI. find
  33. Lambek, Joachim. 2001. Type grammars as pregroups. Grammars, 4(1):21–39. find
  34. Landauer, Thomas K. and Susan T. Dumais. 1997. A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction and representation of knowledge. Psychological Review, 104(2):211–240. find
  35. Lee, Lillian. 1999. Measures of distributional similarity. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL-1999), pages 23–32, College Park, MD. find our card
  36. Lin, Dekang. 1998. Automatic retrieval and clustering of similar words. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics (COLING-ACL ’98), pages 768–774, Montreal. find our card
  37. Lin, Dekang. 2003. Dependency-based evaluation of MINIPAR. In Anne Abeillé, editor, Treebanks: Building and Using Parsed Corpora, pages 317–330, Kluwer, Dordrecht. find
  38. Lodhi, Huma, Craig Saunders, John Shawe-Taylor, Nello Cristianini, and Chris Watkins. 2002. Text classification using string kernels. Journal of Machine Learning Research, 2:419–444. find
  39. McCarthy, Diana, Rob Koeling, Julie Weeds, and John Carroll. 2004. Finding predominant word senses in untagged text. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), pages 279–286, Barcelona. find our card
  40. Miller, George A. and Walter G. Charles. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1):1–28. find
  41. Mitchell, Jeff and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of ACL-08: HLT, pages 236–244, Columbus, OH. find
  42. Preller, Anne and Mehrnoosh Sadrzadeh. 2011. Bell states and negative sentences in the distributed model of meaning. Electronic Notes in Theoretical Computer Science, 270(2):141–153. find
  43. Rudolph, Sebastian and Eugenie Giesbrecht. 2010. Compositional matrix-space models of language. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 907–916, Uppsala. find
  44. Schütze, Heinrich. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97–123. find
  45. Smolensky, Paul. 1990. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence, 46(1-2):159–216. find
  46. Song, Dawei and Peter D. Bruza. 2003. Towards context-sensitive information inference. Journal of the American Society for Information Science and Technology (JASIST), 54:321–334. find
  47. Voiculescu, Dan-Virgil. 1997. Free Probability Theory. American Mathematical Society, Providence, RI. find
  48. Weeds, Julie, David Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In Proceedings of CoLING 2004, pages 1015–1021, Geneva. find
  49. Widdows, Dominic. 2008. Semantic vector products: Some initial investigations. In Proceedings of the Second Symposium on Quantum Interaction, pages 1–8, Oxford. find
  50. Wittgenstein, Ludwig. 1953. Philosophical Investigations. Macmillan, New York. G. Anscombe, translator. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/clarke-2012-context-theoretic-framework-compositionality-distributional/