Elamur Research⌕
☰
home / papers / 2014 / Is there a place for logic in recognizing textual entailment

Is there a place for logic in recognizing textual entailment

Paper · 2014 · Johan Bos

Summary

The paper asks whether deductive logic is practical for recognizing textual entailment (RTE), deciding if a text T entails a hypothesis H. Bos evaluates Nutcracker, an RTE system he developed over several years and distributed with the C&C tools. The pipeline is:

  • the C&C parser produces Combinatory Categorial Grammar derivations;
  • λ-calculus composition turns these into Discourse Representation Structures with neo-Davidsonian events and fully resolved scope;
  • the DRSs are translated into first-order logic in a way that treats modal and embedded propositions properly;
  • a theorem prover and a finite model builder run side by side on the result.

On RTE-2 and RTE-3, the system finds few proofs, but most of them are correct, so precision is high and recall is low. Adding background axioms raises recall with little loss of precision. These come from WordNet, from modality rules, and from rules learned automatically from positive pairs. Bos concludes that the bottleneck is neither translating text into logic nor prover performance. It is the absence of a systematic way to obtain relevant background knowledge, which makes logic most useful as a high-precision component inside ensemble systems.

Key points

  • Without background axioms, Nutcracker found 61 proofs across the 3,200 RTE-2 and RTE-3 pairs, of which 54 were correct (88.5% precision).
  • Background axioms added recall at nearly unchanged precision:
    • WordNet synonym and hyponym axioms gave 12 extra proofs, 11 of them correct;
    • WordNet plus modality axioms gave 21 extra proofs, 18 correct;
    • axioms generated automatically from positive T–H pairs (Ihsani 2012) gave 52 extra proofs, 46 correct.
  • The supported provers include Vampire, which performs best on RTE problems and is top-ranked in CASC, and Bliksem. The supported model builders are Mace and Paradox.
  • Attitude verbs such as think and believe block entailment of their complement, while factives such as know and regret preserve it. Lexical knowledge about verb and adverb factivity is therefore required.
  • Bos reports never meeting an RTE example where correct quantifier-scope resolution mattered. A naive scope assignment that mostly follows surface order did no harm.

Interesting ideas

  • I1. The prover tries to prove the problem while a model builder searches, in parallel, for a finite model of its negation; whichever succeeds first stops the other, yielding proof, counter-model, or undecided.
  • I2. Neo-Davidsonian events, where the event is an individual and roles are separate predicates, are chosen because they reduce the number of meaning postulates needed for inference.
  • I3. Embedded clauses under attitude verbs are kept in their own modal context instead of being asserted as facts, with a lexicon deciding which verbs and adverbs project their complement.
  • I4. Scope ambiguity is resolved greedily by surface order instead of being kept underspecified, because downstream reasoning needs concrete formulas and the cost in accuracy was negligible.
  • I5. Each knowledge source is evaluated by how many extra proofs it unlocks and how many of those are correct, which keeps precision visible while recall grows.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Checking a question against the world of the text could likewise run a "derive it" search and a "build a consistent world without it" search together, giving explainable yes/no/unknown answers.
  • I2 (frames): Representing VerbNet role frames as event nodes with separate role edges should let dropped or optional arguments still match without per-verb rules.
  • I3 (world): Mova could keep a factivity flag on verb classes so that content under "believe" enters the world only as a holder's belief state, while content under "know" enters as plain fact.
  • I4 (tree): Mova could commit to one deterministic scope or attachment reading early instead of carrying ambiguity forward, unless an evaluation shows the distinction actually matters.
  • I5 (learn): Every new knowledge layer (WordNet, FrameNet, learned rules) could be scored by the extra answers it enables and their correctness, so that a precision-damaging source is caught early.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GroningenGroningen · Netherlands49 works · 4,843 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Johan Bos. 2014. Is there a place for logic in recognizing textual entailment. Linguistic Issues in Language Technology.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bos2014is,
  title = {{Is there a place for logic in recognizing textual entailment}},
  author = {Johan Bos},
  booktitle = {Linguistic Issues in Language Technology},
  year = {2014},
  doi = {10.33011/lilt.v9i.1315},
  url = {https://doi.org/10.33011/lilt.v9i.1315}
}

References

28 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Balduccini, M., C. Baral, and Y. Lierler. 2008. Knowledge representation and question answering. In V. Lifschitz, F. van Harmelen, and B. Porter, eds., Handbook of Knowledge Representation, pages 779–819. Elsevier. find
  2. Beaver, David Ian. 1997. Presupposition. In J. Van Benthem and A. Ter Meulen, eds., Handbook of Logic and Language, chap. 17, pages 939–1008. Elsevier, MIT. find
  3. Blackburn, P. and J. Bos. 2005. Representation and Inference for Natural Language. A First Course in Computational Semantics. CSLI. find
  4. Bos, Johan. 2004. Computational semantics in discourse: Underspecification, resolution, and inference. Journal of Logic, Language and Information 13(2):139–157. ↗
  5. Bos, Johan. 2008. Wide-Coverage Semantic Analysis with Boxer. In J. Bos and R. Delmonte, eds., Semantics in Text Processing. STEP 2008 Conference Proceedings, vol. 1 of Research in Computational Semantics, pages 277–286. College Publications. find
  6. Bos, Johan and Katja Markert. 2005. Recognising textual entailment with logical inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 628–635.
  7. Chierchia, Gennaro and Sally McConnell-Ginet. 1991. Meaning and Grammar. An Introduction to Semantics. The MIT Press. find
  8. Claessen, K. and N. Sörensson. 2003. New techniques that improve mace-style model finding. In P. Baumgartner and C. Fermüller, eds., Model Computation – Principles, Algorithms, Applications (Cade-19 Workshop), pages 11–27. Miami, Florida, USA. find
  9. Clark, Stephen and James R. Curran. 2004. Parsing the WSJ using CCG and Log-Linear Models. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL ’04), pages 104–111. Barcelona, Spain.
  10. Curran, James, Stephen Clark, and Johan Bos. 2007. Linguistically Motivated Large-Scale NLP with C&C and Boxer. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 33–36. Prague, Czech Republic.
  11. Dagan, Ido, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Lecture Notes in Computer Science, vol. 3944, pages 177–190. Is there a place for logic in recognizing textual entailment? / 43 find
  12. De Nivelle, Hans. 1998. A Resolution Decision Procedure for the Guarded Fragment. In Automated Deduction - CADE-15. 15th International Conference on Automated Deduction, pages 191–204. Springer-Verlag Berlin Heidelberg. find
  13. Fellbaum, Christiane, ed. 1998. WordNet. An Electronic Lexical Database. The MIT Press. find
  14. Gamut, L.T.F. 1991. Logic, Language, and Meaning. Volume II. Intensional Logic and Logical Grammar . Chicago and London: The University of Chicago Press. find
  15. Heim, Irene and Angelika Kratzer. 1998. Semantics in Generative Grammar . Malden and Oxford: Blackwell. find
  16. Ihsani, Annisa. 2012. Automatic Induction of Background Knowledge Axioms for Recognising Textual Entailment. Master’s thesis, University of Groningen. find
  17. Kamp, Hans and Uwe Reyle. 1993. From Discourse to Logic; An Introduction to Modeltheoretic Semantics of Natural Language, Formal Logic and DRT . Dordrecht: Kluwer. find
  18. Lin, Dekang and Patrick Pantel. 2001. DIRT—discovery of inference rules from text. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 323–328. find
  19. MacCartney, Bill. 2009. Natural Language Inference. Ph.D. thesis, Stanford University. find
  20. MacCartney, Bill, Trond Grenager, Marie-Catherine de Marne↵e, Daniel Cer, and Christopher D. Manning. 2006. Learning to recognize features of valid textual entailments. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, HLT-NAACL ’06, pages 41–48. Stroudsburg, PA, USA: Association for Computational Linguistics. find
  21. McCune, W. 1998. Automatic Proofs and Counterexamples for Some Ortho-lattice Identities. Information Processing Letters 65(6):285–291. find
  22. McCune, W. and R. Padmanabhan. 1996. Automated Deduction in Equational Logic and Cubic Curves. No. 1095 in Lecture Notes in Computer Science (AI subseries). Springer-Verlag. find
  23. Meyers, A., C. Macleod, R. Yangarber, R. Grishman, L. Barrett, and R. Reeves. 1998. Using nomlex to produce nominalization patterns for information extraction. In Coling-ACL98 workshop Proceedings, The Computational Treatment of Nominals, pages 25–32. Montreal, Canada. find
  24. Minnen, Guido, John Carroll, and Darren Pearce. 2001. Applied morphological processing of english. Journal of Natural Language Engineering 7(3):207–223. ↗
  25. Muskens, Reinhard. 1996. Combining Montague Semantics and Discourse Representation. Linguistics and Philosophy 19:143–186. ↗
  26. Riazanov, A. and A. Voronkov. 2002. The Design and Implementation of Vampire. AI Communications 15(2–3):91–110. 44 / Johan Bos Steedman, Mark. 2001. The Syntactic Process. The MIT Press. find
  27. Sutcli↵e, Geo↵ and Christian Suttner. 1997. The results of the cade-13 atp system competition. Journal of Automated Reasoning 18(2):259–264. Special Issue on the CADE-13 Automated Theorem Proving System Competition. find
  28. Szpektor, Idan, Eyal Shnarch, and Ido Dagan. 2007. Instance-based evaluation of entailment rule acquisition. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 456–463. Prague, Czech Republic: Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bos-2014-is-there-place-logic-recognizing/