Elamur Research⌕
☰
home / papers / 2016 / Representing Meaning with a Combination of Logical and Distributional Models

Representing Meaning with a Combination of Logical and Distributional Models

Paper · 2016 · I. Beltagy, Stephen Roller, Pengxiang Cheng et al.

Summary

NLP applications need more from semantics than any single representation offers. Logical forms handle sentence structure, negation and quantifiers, but they are binary and depend on hand-built lexicons. Distributional vectors capture graded similarity between words and phrases but lack structural detail. Beltagy et al. describe a complete hybrid system that links the two through probabilistic logic, specifically Markov Logic Networks (MLNs).

The system works in three steps:

  • Logical representation: sentences become first-order formulas, produced with Boxer.
  • Knowledge base: weighted inference rules are built from distributional models, WordNet and PPDB. A logic-based alignment (a modified Robinson resolution) selects which lexical and phrasal rules are needed.
  • Inference: MLN problems are reformulated so they stay tractable.

For textual entailment, the authors add a closed-world assumption, coreference handling for contradictions, multiple parses, a supervised entailment rule classifier and weight learning. On SICK, the system achieves state-of-the-art accuracy. Because the logic handles sentence structure, the setup also serves as a clean testbed for distributional lexical entailment.

Key points

  • Result: evaluated on SICK's test portion, the full system scores 85.1% accuracy. Of all its components, the classifier for entailment rules (eclassif) matters most.
  • Other knowledge sources: adding PPDB gave no gain, and WordNet slightly reduced accuracy.
  • Soft rules: keeping rules soft and learning their weights helped, even though SICK does not grade entailment. The log-based variant (wlearn_log) was slightly better.
  • Released rule set: 12,510 rules in total, of which 10,211 are unique (3,106 entailing, 177 contradictions, 6,928 neutral). The abstract instead gives 10,213. The data is at github.com/ibeltagy/rrr; the system code is at github.com/ibeltagy/pl-semantics.
  • Distributional evaluation: this is the first test of the question posed by Levy et al. (2015) on real RTE data, and it finds distributional features useful for lexical entailment. The study also evaluates the compositional model of Paperno et al. (2014) on phrasal entailment. It notes that cosine similarity wrongly favors co-hyponyms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States109 works · 56,942 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as I. Beltagy, Stephen Roller, Pengxiang Cheng, Katrin Erk and Raymond J. Mooney. 2016. Representing Meaning with a Combination of Logical and Distributional Models. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00266

@article{beltagy2016representing,
  title = {{Representing Meaning with a Combination of Logical and Distributional Models}},
  author = {I. Beltagy, Stephen Roller, Pengxiang Cheng, Katrin Erk and Raymond J. Mooney},
  journal = {Computational Linguistics},
  year = {2016},
  doi = {10.1162/coli\_a\_00266},
  url = {https://doi.org/10.1162/coli_a_00266}
}

References

90 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. The Core Language Engine. MIT Press. Andrews, Mark, Gabriella Vigliocco, and David Vinson. 2009. Integrating experiential and distributional data to learn semantic representations. Psychological Review, 116(3):463–498. find
  2. Bach, Stephen H., Bert Huang, Ben London, and Lise Getoor. 2013. Hinge-loss Markov random fields: Convex inference for structured prediction. In UAI 2013, pages 32–41, Bellevue, WA. find
  3. Baroni, Marco, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012. Entailment above the word level in distributional semantics. In EACL 2012, pages 23–32, Avignon.
  4. Baroni, Marco, Raffaella Bernardi, and Roberto Zamparelli. 2014. Frege in space: A program for compositional distributional semantics. Linguistic Issues in Language Technology, 9(6):5–110. find
  5. Baroni, Marco and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics.
  6. Computational Linguistics, 36(4):673–721. Baroni, Marco and Alessandro Lenci. 2011. How we BLESSed distributional semantic evaluation. In GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pages 1–10, Edinburgh.
  7. Baroni, Marco and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In EMNLP 2010, pages 1183–1193, Cambridge, MA. find
  8. Beltagy, I., Cuong Chau, Gemma Boleda, Dan Garrette, Katrin Erk, and Raymond Mooney. 2013. Montague meets Markov: Deep semantics with probabilistic logical form. In *SEM 2013, pages 11–21, Atlanta, GA. find
  9. Beltagy, I. and Katrin Erk. 2015. On the proper treatment of quantifiers in probabilistic logic semantics. In IWCS 2015, pages 140–150, London. find
  10. Beltagy, I., Katrin Erk, and Raymond Mooney. 2014. Probabilistic soft logic for semantic textual similarity. In ACL 2014, pages 1210–1219, Baltimore, MD. find
  11. Beltagy, I. and Raymond J. Mooney. 2014. Efficient Markov logic inference for natural language semantics. In StarAI 2014, pages 9–14. find
  12. Quebec City. Berant, Jonathan, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In EMNLP 2013, pages 1533–1544, Seattle, WA.
  13. Bird, Steven, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. find
  14. O’Reilly Media, Inc. Bjerva, Johannes, Johan Bos, Rob van der Goot, and Malvina Nissim. 2014. The meaning factory: Formal semantics for recognizing textual entailment and determining semantic similarity. In SemEval 2014, pages 642–646, Dublin.
  15. Blackburn, Patrick and Johan Bos. 2005. Representation and Inference for Natural Language: A First Course in Computational Semantics. find
  16. CSLI Publications, Stanford, CA. Bos, Johan. 2008. Wide-coverage semantic analysis with Boxer. In Proceedings of Semantics in Text Processing (STEP-2008), pages 277–286, Venice. find
  17. Bos, Johan. 2009. Applying automated deduction to natural language understanding. Journal of Applied Logic, 7:100–112. find
  18. Broecheler, Matthias, Lilyana Mihalkova, and Lise Getoor. 2010. find
  19. Probabilistic Similarity Logic. In UAI 2010, pages 73–82, Catalina Island, CA. find
  20. Brown, Susan Windisch. 2008. Choosing sense distinctions for WSD: Psycholinguistic evidence. In ACL 2008, pages 249–252, Columbus, OH. find
  21. Bruni, Elia, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. Distributional semantics in technicolor. In ACL 2012, pages 136–145, Jeju Island.
  22. Bunescu, Razvan and Ray Mooney. 2007. Multiple instance learning for sparse positive bags. In ICML 2007, pages 105–112, Corvalis. find
  23. Chang, Chih-Chung and Chih-Jen Lin. 2001. LIBSVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/~cjlin/ libsvm. ↗
  24. Clark, Stephen. 2012. Vector space models of lexical meaning. In J. van Bentham, M. Moortgat, and W. Buszkowski, editors, Handbook of Contemporary Semantics. Wiley-Blackwell, 2 edition, pages 345–384. find
  25. Clark, Stephen and James R. Curran. 2004. Parsing the WSJ using CCG and log-linear models. In ACL 2004, pages 103–110, Barcelona.
  26. Coecke, Bob, Mehrnoosh Sadrzadeh, and Stephen Clark. 2011. Mathematical foundations for a compositional distributed model of meaning. Linguistic Analysis, 36(1-4):345–384. find
  27. Cooper, Robin, Simon Dobnik, Shalom Lappin, and Staffan Larsson. 2015. Probabilistic type theory and natural language semantics. Linguistic Issues in Language Technology, 10(4):1–43. find
  28. Copestake, Ann and Dan Flickinger. 2000. An open-source grammar development environment and broad-coverage English grammar using HPSG. In LREC 2000, Athens. find
  29. Dagan, Ido, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4):1–220. find
  30. Dechter, Rina, Kalev Kask, and Robert Mateescu. 2002. Iterative join-graph propagation. In UAI 2002, pages 128–136, Edmonton. find
  31. Dietterich, Thomas G., Richard H. Lathrop, and Tomas Lozano-Perez. 1997. Solving the multiple instance problem with axis-parallel rectangles. Artificial Intelligence, 89(1-2):31–71. Dinu, Georgiana, Stefan Thater, and Sören Laue. 2012. A comparison of models of word meaning in context. In HLT-NAACL… find
  32. Dowty, David R., Robert E. Wall, and Stanley Peters. 1981. Introduction to Montague Semantics. find
  33. D. Reidel, Dordrecht, Holland. Edmonds, Philip and Graeme Hirst. 2000. Reconciling fine-grained lexical knowledge and coarse-grained ontologies in the representation of near-synonyms. In Proceedings of the Workshop on Semantic Approximation, Granularity, and Vagueness, Breckenridge, CO. Erk, Katrin… find
  34. Fu, Ruiji, Jiang Guo, Bing Qin, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014. Learning semantic hierarchies via word embeddings. In ACL 2014, pages 1199–1209, Baltimore, MD.
  35. Ganitkevitch, Juri, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In NAACL-HLT 2013, pages 758–764, Atlanta, GA.
  36. Garrette, Dan, Katrin Erk, and Raymond Mooney. 2011. Integrating logical representations with probabilistic information using Markov logic. In IWCS 2011, pages 105–114, Oxford. find
  37. Geffet, Maayan and Ido Dagan. 2005. The distributional inclusion hypotheses and lexical entailment. In ACL 2005, pages 107–114, Ann Arbor, MI.
  38. Genesereth, M. R. and N. J. Nilsson. 1987. Logical foundations of artificial intelligence. find
  39. Morgan Kaufman, San Mateo, CA. Getoor, L. and B. Taskar. 2007. Introduction to Statistical Relational Learning. find
  40. MIT Press, Cambridge, MA. Geurts, Bart. 2007. Existential import. In I. Comorovski and K. van Heusinger, editors, Existence: syntax and semantics. Springer, Dordrecht, pages 253–271. find
  41. Gogate, Vibhav. 2014. IJGP-sampling and SampleSearch. Software available at http://www.hlt.utdallas.edu/~vgogate/ ijgp-samplesearch.html. ↗
  42. Gogate, Vibhav and Rina Dechter. 2011. SampleSearch: Importance sampling in presence of determinism. Artificial Intelligence, 175(2):694–729. find
  43. Gogate, Vibhav and Pedro Domingos. 2011. Probabilistic theorem proving. In UAI 2011, pages 256–265, Barcelona. find
  44. Grefenstette, Edward. 2013. Towards a formal distributional semantics: Simulating logical calculi with tensors. In *SEM 2013, pages 11–21, Atlanta, GA. find
  45. Grefenstette, Edward and Mehrnoosh Sadrzadeh. 2011. Experimental support for a categorical compositional distributional model of meaning. In EMNLP 2011, pages 1394–1404, Edinburgh, Scotland. Herbelot, Aurélie and Eva Maria Vecchi. 2015. Building a shared world: Mapping distributional to model-theor… find
  46. Hobbs, Jerry R., Mark E. Stickel, Paul Martin, and Douglas Edwards. 1988. Interpretation as abduction. In ACL 1988, pages 95–103, Buffalo, NY. find
  47. Kamp, Hans and Uwe Reyle. 1993. From Discourse to Logic. find
  48. Kluwer. Kok, Stanley, Parag Singla, Matthew Richardson, and Pedro Domingos. 2005. The Alchemy system for statistical relational AI. http://www.cs.washington.edu/ai/alchemy. ↗
  49. Kotlerman, Lili, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-Geffet. 2010. Directional distributional similarity for lexical inference. Natural Language Engineering, 16(4):359–389. find
  50. Kruszewski, German, Denis Paperno, and Marco Baroni. 2015. Deriving Boolean structures from distributional vectors.
  51. Kwiatkowski, Tom, Eunsol Choi, Yoav Artzi, and Luke Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In EMNLP 2013, pages 1545–1556, Seattle, WA.
  52. Lai, Alice and Julia Hockenmaier. 2014. Illinois-lh: A denotational and distributional approach to semantics. In SemEval 2014, pages 329–334, Dublin. find
  53. Landauer, Thomas and Susan Dumais. 1997. A solution to Plato’s problem: The Latent Semantic Analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211. find
  54. Lenci, Alessandro and Giulia Benotto. 2012. Identifying hypernyms in distributional semantic spaces. In *SEM 2012, pages 75–79, Montreal.
  55. Levy, Omer, Steffen Remus, Chris Biemann, and Ido Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? In NAACL HLT 2015, pages 970–976, Denver, CO.
  56. Lewis, Mike and Mark Steedman. 2013. Combined distributional and logical semantics. find
  57. Lewis, Mike and Mark Steedman. 2014. A* CCG parsing with a supertag-factored model. In EMNLP 2014, pages 990–1000, Doha.
  58. Liang, Percy, Michael Jordan, and Dan Klein. 2011. Learning dependency-based compositional semantics. In ACL-HLT 2011, pages 590–599, Portland, OR. find
  59. Lund, Kevin and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, and Computers, 28(2):203–208. find
  60. MacCartney, Bill and Christopher D. Manning. 2009. An extended model of natural logic. In IWCS 2009, pages 140–156, Tilburg. find
  61. Marelli, Marco, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014a. SemEval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In SemEval 2014, pages 1–8, Dublin…
  62. Marelli, Marco, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014b. A SICK cure for the evaluation of compositional distributional semantic models. In LREC 2014, pages 216–223, Reykjavik.
  63. Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In ICLR 2013, Scottsdale, AZ.
  64. Mikolov, Tomas, Wentau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In NAACL-HLT 2013, pages 746–751, Atlanta, GA.
  65. Mitchell, Jeff and Mirella Lapata. 2008. Vector-based models of semantic composition. In ACL 2008, pages 236–244, Columbus, OH. find
  66. Mitchell, Jeff and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(3):1388–1429. find
  67. Montague, Richard. 1970. Universal grammar. Theoria, 36:373–398. find
  68. Ng, Dominick and James R Curran. 2012. Dependency hashing for n-best CCG parsing. In ACL 2012, pages 497–505, Jeju Island. find
  69. Paperno, Denis, Nghia The Pham, and Marco Baroni. 2014. A practical and linguistically-motivated approach to compositional distributional semantics. In ACL 2014, pages 90–99, Baltimore, MD. find
  70. Parsons, Terry. 1990. Events in the semantics of English. find
  71. MIT Press, Cambridge, MA. Pasupat, Panupong and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In ACL 2015, pages 1470–1480, Beijing.
  72. Pearl, Judea. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. find
  73. Morgan Kaufmann, San Mateo, CA. Poon, Hoifung and Pedro Domingos. 2008. Joint unsupervised coreference resolution with Markov logic. In EMNLP 2008, pages 650–659, Honolulu, HI.
  74. Princeton University. 2010. About WordNet. http://wordnet.princeton.edu. ↗
  75. Richardson, Matthew and Pedro Domingos. 2006. Markov logic networks. Machine Learning, 62:107–136. find
  76. Riedel, Sebastian, Hong-Woo Chun, Toshihisa Takagi, and Jun’ichi Tsujii. 2009. A Markov logic approach to bio-molecular event extraction. In BioNLP 2009, pages 41–49, Boulder, CA. find
  77. Riedel, Sebastian and Ivan Meza-Ruiz. 2008. Collective semantic role labelling with Markov logic. In CoNLL’08, pages 193–197, Manchester. find
  78. Robinson, J. A. 1965. A machine-oriented logic based on the resolution principle. Journal of the ACM, 12(1):23–41. find
  79. Roller, Stephen, Katrin Erk, and Gemma Boleda. 2014. Inclusive yet selective: Supervised distributional hypernymy detection. In COLING 2014, pages 1025–1036, Dublin.
  80. Sadrzadeh, Mehrnoosh, Stephen Clark, and Bob Coecke. 2013. The Frobenius anatomy of word meanings I: subject and object relative pronouns. Journal of Logic and Computation, 23(6):1293–1317. find
  81. Santus, Enrico. 2013. SLQS: An entropy measure. Master’s thesis, University of Pisa. Schütze, Hinrich. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97–123.
  82. Silberer, Carina and Mirella Lapata. 2012. Grounded models of semantic representation. In EMNLP-CoNLL 2012, pages 1423–1433, Jeju Island. find
  83. Skolem, Thoralf. 1920. Logisch-kombinatorische Untersuchungen über die Erfüllbarkeit oder Beweisbarkeit mathematischer Sätze. Skrifter utgit av Videnskapselskapet i Kristiania, 4:4–36. find
  84. Snow, Rion, Daniel Jurafsky, and Andrew Y. Ng. 2006. Semantic taxonomy induction from heterogenous evidence. In COLING/ ACL 2006, pages 801–808, Sydney. find
  85. Strawson, P. F. 1950. On referring. Mind, 59:320–344. Thater, Stefan, Hagen Fürstenau, and Manfred Pinkal. 2010. Contextualizing semantic representations using syntactically enriched vector models. In ACL 2010, pages 948–957, Uppsala. find
  86. Tian, Ran, Yusuke Miyao, and Matsuzaki Takuya. 2014. Logical inference on dependency-based compositional semantics. In ACL 2014, pages 78–89, Baltimore, MD. find
  87. Turney, Peter and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37(1):141–188. van Eijck, Jan and Shalom Lappin. 2012. Probabilistic semantics for natural language. In Logic and interactive rationality (LIRA) yearbook.…
  88. Weeds, Julie, Daoud Clarke, Jeremy Reffin, David Weir, and Bill Keller. 2014. Learning to distinguish hypernyms and co-hyponyms. In COLING 2014, pages 2249–2259, Dublin.
  89. Weeds, Julie, David Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In COLING 2004, pages 1015–1021, Geneva. find
  90. Zelle, John M. and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In AAAI 1996, pages 1050–1055, Portland, OR. Zirn, Cäcilia, Mathias Niepert, Heiner Stuckenschmidt, and Michael Strube. 2011. Fine-grained sentiment analysis with structural features. I…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/beltagy-2016-representing-meaning-combination-logical-distributional/