Elamur Research⌕
☰
home / papers / 2007 / A Discourse Commitment-Based Framework for Recognizing Textual Entailment

A Discourse Commitment-Based Framework for Recognizing Textual Entailment

Paper · 2007 · Andrew Hickl and Jeremy Bensley

Summary

Hickl and Bensley tackle Recognizing Textual Entailment (RTE). They argue that shallow overlap or graph-alignment methods will break down as text–hypothesis pairs grow longer and more complex. Their alternative first spells out the discourse commitments of each passage: the propositions a conventional reading forces a reader to accept, whether asserted, presupposed or implicated. Entailment is then decided over pairs of these short commitments.

The pipeline has four stages:

  • heavy preprocessing: Collins-style parsing, PropBank/NomBank semantic dependencies, NER, coreference and temporal/spatial normalization;
  • heuristic commitment extraction;
  • alignment-based selection of the text commitment that best matches each hypothesis commitment;
  • a decision-tree entailment classifier, whose positive verdicts are rechecked by a contradiction detector.

Trained only on the 800 RTE-3 Development pairs, with hand-made token alignments, the system classified just over 80% of the RTE-3 Test Set correctly. It used no web resources or extra corpora. Performance held up on long texts. This suggests that explicitly listing inferences reduces the need for massive training data.

Key points

  • Commitments come from five kinds of heuristics:
    • sentence segmentation;
    • splitting coordinations and lists;
    • turning supplemental expressions (appositives, parentheticals, non-restrictive relatives, epithets, as-clauses) into standalone conventional-implicature sentences;
    • six types of named-entity relations;
    • coreference-driven re-generation.
  • Commitment selection uses the Taskar et al. large-margin word-alignment model with string, WordNet-similarity and Dice-coefficient features. It scored:
    • token-alignment precision 94.55% (MRR 0.9219);
    • commitment-selection precision 89.50%, up from 72.50% for the earlier Hickl et al. system.
  • The official run reached 0.8038 accuracy. Short and long pairs scored about the same (0.8023 vs 0.8120).
  • Contradiction-based validation helped only modestly:
    • without it, accuracy was 0.7865;
    • with the default λ=0.85, it was 0.8038;
    • with λ=0.75, tuned post hoc on the test set, it was 0.8125.
  • Adding training data did not keep helping. Accuracy peaked at 25,000 pairs (0.8225), and with 100,000 pairs it fell back to 0.8050 while average precision dropped to 0.8003.

Interesting ideas

  • I1. Break a sentence into many tiny self-contained propositions before comparing meanings, so that matching becomes short-against-short rather than long-against-short.
  • I2. Treat appositives, parentheticals and non-restrictive relative clauses as carrying conventional implicatures that stay true regardless of the main clause, and rewrite each as its own statement.
  • I3. Multiply commitments by substituting each coreferent mention (name, role noun, epithet) into existing propositions, so that later matching finds a surface form that lines up.
  • I4. Pick the matching pair by reciprocal best hit: candidate A must be B's top match and B must be A's top match. Then try lower-scoring pairs in order until one passes or a threshold is reached.
  • I5. Run a separate contradiction check over all remaining text propositions after a positive entailment decision, and veto the yes if anything conflicts.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): The world-of-text builder could emit an explicit list of atomic facts per sentence, including those from appositives and relative clauses, as the checkable basis for answers.
  • I2 (frames): prag could detect these UD constructions (appos, acl:relcl with commas, parataxis) and add separate frames marked as presupposed or not-at-issue content.
  • I3 (frames): After coreference, the frames could be indexed under all mentions of an entity, letting the answer step match a question that uses a different description.
  • I4 (answer): Readers aligning a question to text facts could require mutual best matches before committing, giving a simple explainable filter against spurious alignments.
  • I5 (answer): A final consistency pass over the world state could catch answers supported by one fact but contradicted by another, and report the conflicting fact as the explanation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Language ComputerRichardson · United States9 works · 1,157 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksDiscourse analysis · 466 works

Cite

Cite as Andrew Hickl and Jeremy Bensley. 2007. A Discourse Commitment-Based Framework for Recognizing Textual Entailment. Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hickl2007discourse,
  title = {{A Discourse Commitment-Based Framework for Recognizing Textual Entailment}},
  author = {Andrew Hickl and Jeremy Bensley},
  booktitle = {Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing},
  year = {2007},
  doi = {10.3115/1654536.1654571},
  url = {https://doi.org/10.3115/1654536.1654571}
}

References

23 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Paul Aarseth, John Lehmann, Murat Deligonul, and Luke Nezda. 2006. TASER: A Temporal and Spatial Expression Recognition and Normalization System. In Proceedings of the Automatic Content Extraction (ACE) Conference. find
  2. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The Second PASCAL Recognising Textual Entailment Challenge. In Proceedings of the Second PASCAL Challenges Workshop. find
  3. Johan Bos and Katya Markert. 2006. When logical inference helps in determining textual entailment (and when it doesn’t). In Proceedings of the Second PASCAL Recognizing Textual Entailment Conference, Venice, Italy. find
  4. Michael Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, Univ. of Pennsylvania. find
  5. L.R. Dice. 1945. Measures of the Amount of Ecologic Association Between Speices. In Journal of Ecology, volume 26, pages 297–302. find
  6. Oren Glickman and Ido Dagan. 2005. A Probabilistic Setting and Lexical Co-occurrence Model for Textual Entailment. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, Ann Arbor, USA. find
  7. Christine Gunlogson. 2001. True to Form: Rising and Falling Declaratives as Questions in English. Ph.D. thesis, University of California, Santa Cruz. find
  8. Aria Haghighi, Andrew Ng, and Christopher Manning. 2005. Robust textual inference via graph matching. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 387–394. find
  9. Sanda Harabagiu, Andrew Hickl, and Finley Lacatusu. 2006. Negation, Contrast, and Contradiction in Text Processing. In Proceedings of AAAI, Boston, MA. find
  10. Andrew Hickl, John Williams, Jeremy Bensley, Kirk Roberts, Bryan Rink, and Ying Shi. 2006. Recognizing Textual Entailment with LCC’s Groundhog System. In Proceedings of the Second PASCAL Challenges Workshop. find
  11. Rodney Huddleston and Geoffrey Pullum, editors, 2002. The Cambridge Grammar of the English Language. Cambridge-University Press. ↗
  12. V. Jijkoun and M. de Rijke. 2005. Recognizing Textual Entailment Using Lexical Similarity. In Proceedings of the First PASCAL Challenges Workshop. find
  13. X. Luo, A. Ittycheriah, H. Jing, N. Kambhatla, and S. Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the Bell Tree. In Proceedings of the ACL-2004, Barcelona, Spain.
  14. A. Meyers, R. Reeves, C. Macleod, R. Szekely, V. Zielinska, B. Young, and R. Grishman. 2004. The nombank project: An interim report. In A. Meyers, editor, HLT-NAACL 2004 Workshop: Frontiers in Corpus Annotation, pages 24–31, Boston, Massachusetts, USA, May 2 - May 7. Association for Computational Linguistics.
  15. Arcady Mushegian and Eugene Koonin. 2005. A minimal gene set for cellular life derived by compraison of complete bacterial genomes. In Proceedings of the National Academies of Science, volume 93, pages 10268–10273. find
  16. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An Annotated Corpus of Semantic Roles. Computational Linguistics, 31(1):71–106. ↗
  17. T. Pedersen, S. Patwardhan, and J. Michelizzi. 2004. Word-Net::Similarity - Measuring the Relatedness of Concepts. In Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI-04), San Jose, CA. find
  18. Christopher Potts, editor, 2005. The Logic of Conventional Implicatures. Oxford University Press. ↗
  19. R. Quinlan. 1998. C5.0: An Informal Tutorial. RuleQuest. find
  20. Robert Stalnaker, 1979. Assertion, volume 9, pages 315–332. find
  21. Ben Taskar, Simone Lacoste-Julien, and Michael Jordan. 2005a. Structured prediction via the extragradient method. In Proceedings of Neural Information Processing Systems, Vancouver, Canada. find
  22. Ben Taskar, Simone Lacoste-Julien, and Dan Klein. 2005b. A discriminative matching approach to word alignment. In Proceedings of Human Language Technology Conference and Empirical Methods in Natural Language Processing (HLT/EMNLP 2005), Vancouver, Canada.
  23. Lucy Vanderwende, Arul Menezes, and Rion Snow. 2006. Microsoft Research at RTE-2: Syntactic Contributions in the Entailment Task: an implementation. In Proceedings of the Second PASCAL Challenges Workshop. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hickl-2007-discourse-commitment-based-framework-recognizing/