Recognising Textual Entailment with Logical Inference
What it is about
The paper compares two domain-independent methods for recognising textual entailment on the RTE challenge data (567 development and 800 test pairs). The shallow method uses word overlap extended by WordNet synonyms and derivations, weighted by inverse document frequency. The deep method parses text and hypothesis with a CCG parser into discourse representations, translates them into first-order logic, and asks a theorem prover (Vampire) and a model builder (Paradox), with background axioms from WordNet and the CIA Factbook. Entailment is approximated by how much the model's domain grows when the hypothesis is added. A decision tree combines the features. On the test set the shallow method reaches 56.9%, the prover alone about 52%, the deep method with model building 56.2%, and the hybrid with the RTE task type as a feature 61.2%. The prover finds only 30 proofs, so its recall is 0.058 at precision 0.767; missing background knowledge is the main cause.
Ideas
- I1. Domain-size difference as a counting feature. Count the entities in the world built from the text and from text plus hypothesis; the absolute and relative growth are numbers Mova can explain ("the hypothesis adds two entities"). Test as a decision-list feature on the development set, run twice.
- I2. Task type as a feature. The paper gains about three points by adding the subset of the RTE data as input. A Mova rule combiner can take the lesson type the same way, chosen on cross-validation, with the test set touched once.
- I3. Explicit lexical rules: hyponym pairs entail, sister terms contradict. Keep them as a small lexicon in
en/src/dict.rs, and log each rule that fires. - I4. Report precision and recall of the proof route separately, use the proof only when it fires, and let the approximation handle the rest.
For Mova
Chapter: languages/en/27-negation.md (negated hypotheses, the paper's Example 78 type). Code: prag/src/infer.rs for the decision, en/src/dict.rs for the lexical relations, world/src/qa.rs for the school questions. Measure: accuracy by route (proof, approximation, both), run twice for identical output.
Library summary
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/H05-1079.pdf
Summary
The paper addresses recognising textual entailment (RTE), which means deciding whether a hypothesis H follows from a text T. The authors test their methods on the PASCAL RTE challenge corpus. They compare two families of methods:
- Shallow method: IDF-weighted lexical overlap that uses WordNet synonymy and derivations, plus length features.
- Deep method: a CCG parser maps T and H to Discourse Representation Structures, which are translated to first-order logic and passed to the Vampire theorem prover and the Paradox model builder.
Proofs rarely succeed because background knowledge is missing. To compensate, the authors use model building: comparing the sizes of the models Paradox finds for T alone and for T+H gives a graded signal of entailment. Machine learning then combines the shallow and deep features.
The two methods reach similar overall accuracy but behave quite differently across the corpus subsets. Merging them alone gives no significant gain. Adding the subset type as a feature raises accuracy to 0.612, which is above the reported range of RTE workshop systems. The study shows that model building can soften the knowledge bottleneck of logic-based inference. It also shows that how a dataset was built strongly shapes which method works best.
Key points
- Background knowledge: the deep pipeline draws on about 20 hand-written generic axioms, WordNet hyponymy and sister-synset axioms, and geographic facts extracted from the CIA factbook. WordNet senses are chosen with a Lesk-style WSD variant.
- Strengths and weaknesses of the deep method: proofs alone gave high precision but low recall. Paradox model sizes served as a robust approximation of entailment where no proof could be found.
- Subset behaviour: the shallow classifier is useful mainly on the CD subset. The deep classifier does comparatively better on several other subsets, and the plain hybrid is more stable across subsets.
- Best result: the Hybrid+Task system, using the subset indicator (CD, IE, MT, RC, IR, PP, QA) as a feature, reached accuracy 0.612 and cws 0.646. RTE workshop systems ranged from 0.50 to 0.59 in accuracy and 0.50 to 0.69 in cws.
- Future work: the authors recommend automatic acquisition of paraphrase-based axioms and task-based evaluation inside QA or summarisation systems.
Cite
@inproceedings{bos2005recognising,
title = {{Recognising Textual Entailment with Logical Inference}},
author = {Bos, Johan and Markert, Katja},
booktitle = {Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing},
year = {2005},
doi = {10.3115/1220575.1220654},
url = {https://doi.org/10.3115/1220575.1220654}
}References
- E. Akhmatova. 2005. Textual entailment resolution via atomic propositions. In PASCAL. Proc. of the First Challenge Workshop. Recognizing Textual Entailment. find
- S. Bayer, J. Burger, L. Ferro, J. Henderson, and A. Yeh. 2005. Mitre’s submission to the eu pascal rte challenge. In PASCAL. Proc. of the First Challenge Workshop. Recognizing Textual Entailment. find
- P. Blackburn and J. Bos. 2005. Representation and Inference for Natural Language. A First Course in Computational Semantics. CSLI. find
- J. Bos, S. Clark, M. Steedman, J. Curran, and J. Hockenmaier. 2004. Wide-coverage semantic representations from a CCG parser. In Proc of COLING. find
- K. Claessen and N. Sörensson. 2003. New techniques that improve mace-style model finding. In Model Computationa - Principles, Algorithms, Applications (Cade-19 Workshop), Miami, Florida. find
- I. Dagan, O. Glickman, and B. Magnini. 2005. The pascal recognising textual entailment challenge. In PAS-CAL. Proc. of the First Challenge Workshop. Recognizing Textual Entailment. find
- C. Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, Mass. find
- A. Fowler, B. Hauser, D. Hodges, I. Niles, A. Novischi, and J. Stephan. 2005. Applying cogex to recognize textual entailment. In PASCAL. Proc. of the First Challenge Workshop. Recognizing Textual Entailment. find
- H. Kamp and U. Reyle. 1993. From Discourse to Logic. Introduction to Modeltheoretic Semantics of Natural Language, Formal Logic and Discourse Representation Theory. Kluwer, Dordrecht, Netherlands. find
- C. Manning and H. Schuetze. 1999. Foundations of Statistical Natural Language Processing. MIT Press. find
- C. Monz and M. de Rijke. 2003. Light-weight entailment checking for computational semantics. In Proc. of ICOS-3. find
- R. Raina, A.Y. Ng, and C. Manning. 2005. Robust textual inference via learning and abductive reasoning. In Proc. of AAAI 2005. find
- A. Riazanov and A. Voronkov. 2002. The design and implementation of Vampire. AI Comm., 15(2-3). find
- H. Saggion, R. Gaizauskas, M. Hepple, I. Roberts, and M Greenwood. 2004. Exploring the performance of boolean retrieval strategies for open domain question answering. In Proc. of the IR4QA Workshop at SIGIR. find
- I. H. Witten and E. Frank. 2000. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, San Diego, CA. find