Elamur Research⌕
☰
home / papers / 2009 / Inference Rules and their Application to Recognizing Textual Entailment

Inference Rules and their Application to Recognizing Textual Entailment

Paper · 2009 · Georgiana Dinu and Rui Wang

Summary

The paper looks at how an automatically mined collection of inference rules can be improved and then used for Recognizing Textual Entailment (RTE). The authors start from DIRT, a large set of paraphrase-like rules learned with the Extended Distributional Hypothesis. Each DIRT rule links two Minipar dependency paths that connect noun slots X and Y. WordNet is used in two ways:

  • Extension: new rules are created by replacing words inside rule patterns with their WordNet synonyms.
  • Filtering: rules are removed when their two sides are identical except for an antonym pair, as in having versus lacking confidence.

To limit where rules fire, the authors build a Tree Skeleton from the dependency parses of T and H. This is a root with left and right dependency paths leading to shared anchors. Rules are matched only between corresponding paths or roots. Coverage is small, since skeletons exist for only 37% of RTE-2 test pairs. On the pairs it does cover, however, the method is clearly more precise than a word-overlap baseline. The broader point is that a rule resource pays off only when a structure fixes the context in which each rule applies.

Key points

  • DIRT was chosen because it was the largest available rule collection, with roughly 50% accuracy on its top paraphrases (per Szpektor et al., 2007).
  • Tree skeletons were extracted for 296 of the 800 RTE-2 test pairs (37%), and other datasets behaved similarly.
  • Coverage/precision results on tree skeletons:
    • DIRT alone: 49 pairs/69.38% on RTE2 and 42 pairs/69.04% on RTE3.
    • Dirt + Id + WN: 130/65.38% and 93/72.05%.
    • The same rules applied without tree skeletons: 673/50.07% and 661/55.06%.
  • On covered pairs, the main setup reaches 60.00% vs. 51.76% for BoW on RTE2 (85 pairs) and 62.50% vs. 54.68% on RTE3 (64 pairs). On the full data, the gains are small: 56.87→57.75% and 61.12→61.75%.
  • Of the 25 misclassified RTE3 pairs, 32% come from wrong rule application, 16% from incorrect rules and 52% from other phenomena (e.g. an imitation diamond is not a diamond).

Interesting ideas

  • I1. Remove a learned rule whose two patterns match in edge labels and content words except for an antonym pair; by construction, nearly all such rules are contradictions, so the filter is cheap and high-precision.
  • I2. Multiply a seed rule by substituting WordNet synonyms for each lexical item in its patterns, which turns one mined rule into many variants.
  • I3. Fire a rule only between structurally corresponding positions (root to root, left path to left path); in the paper this raised precision from about 50–55% to 65–72%.
  • I4. Let a narrow, precise method override a broad baseline only on the items it covers, and report coverage and precision separately.
  • I5. Embedding verbs such as threaten, manage, forget and attempt change whether the embedded event actually happened, even when the embedded verbs match perfectly.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Run this check while compiling paraphrase or verb-relation knowledge, so that "have X" ≈ "lack X" style entries never reach the knowledge graph.
  • I2 (world): Expand frame-level equivalences (e.g. face threat ≈ at risk) via WordNet synsets, keeping the origin of each variant so derived rules stay explainable.
  • I3 (answer): When checking whether an answer follows from a text, apply paraphrase rules only between aligned predicate–argument slots of the two UD trees, not anywhere in the sentence.
  • I4 (learn): Evaluate each new reasoning component as a coverage/precision pair against a bag-of-words negative control, so a small but accurate module is not hidden by full-set averages.
  • I5 (frames): Mark frames under such embedding verbs with factuality (happened / intended / failed) before the world model treats the event as real.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany148 works · 18,287 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Georgiana Dinu and Rui Wang. 2009. Inference Rules and their Application to Recognizing Textual Entailment. Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{dinu2009inference,
  title = {{Inference Rules and their Application to Recognizing Textual Entailment}},
  author = {Georgiana Dinu and Rui Wang},
  booktitle = {Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009)},
  year = {2009},
  doi = {10.3115/1609067.1609090},
  url = {https://doi.org/10.3115/1609067.1609090}
}

References

19 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roy Bar-Haim, Ido Dagan, Iddo Greental, Idan Szpektor, and Moshe Friedman. 2007. Semantic inference at the lexical-syntactic level for textual entailment recognition. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 131–136, Prague, June. Association for Computational Linguistics.
  2. Regina Barzilay and Kathleen R. McKeown. 2001. Extracting paraphrases from a parallel corpus. In Proceedings of 39th Annual Meeting of the Association for Computational Linguistics, pages 50–57, Toulouse, France, July. Association for Computational Linguistics.
  3. Roberto Basili, Diego De Cao, Paolo Marocco, and Marco Pennacchiotti. 2007. Learning selectional preferences for entailment or paraphrasing rules. In In Proceedings of RANLP, Borovets, Bulgaria. find
  4. Peter Clark, Phil Harrison, John Thompson, William Murray, Jerry Hobbs, and Christiane Fellbaum. 2007. On the role of lexical and world knowledge in rte3. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 54–59, Prague, June. Association for Computational Linguistics. find
  5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Lecture Notes in Computer Science, Vol. 3944, Springer, pages 177–190. Quionero-Candela, J.; Dagan, I.; Magnini, B.; d’Alch-Buc, F. Machine Learning Challenges. find
  6. Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. Finding contradictions in text. In Proceedings of ACL-08: HLT, pages 1039–1047, Columbus, Ohio, June. Association for Computational Linguistics.
  7. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1–9, Prague, June. Association for Computational Linguistics.
  8. Adrian Iftene and Alexandra Balahur-Dobrescu. 2007. Hypothesis transformation and semantic variability rules used in recognizing textual entailment. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 125–130, Prague, June. Association for Computational Linguistics.
  9. Dekang Lin and Patrick Pantel. 2001. Dirt. discovery of inference rules from text. In KDD ’01: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pages 323–328, New York, NY, USA. ACM. find
  10. Dekang Lin. 1998. Dependency-based evaluation of minipar. In Proc. Workshop on the Evaluation of Parsing Systems, Granada. find
  11. Erwin Marsi, Emiel Krahmer, and Wauter Bosma. 2007. Dependency-based paraphrasing for recognizing textual entailment. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 83–88, Prague, June. Association for Computational Linguistics.
  12. Rowan Nairn, Cleo Condoravdi, and Lauri Karttunen. 2006. Computing relative polarity for textual inference. In Proceedings of ICoS-5 (Inference in Computational Semantics, Buxton, UK.
  13. Bo Pang, Kevin Knight, and Daniel Marcu. 2003. Syntax-based alignment of multiple translations: Extracting paraphrases and generating new sentences. In HLT-NAACL, pages 102–109.
  14. Satoshi Sekine. 2005. Automatic paraphrase discovery based on context and keywords between NE pairs. In Proceedings of International Workshop on Paraphrase, pages 80–87, Jeju Island, Korea. find
  15. Idan Szpektor, Hristo Tanev, Ido Dagan, and Bonaventura Coppola. 2004. Scaling web-based acquisition of entailment relations. In In Proceedings of EMNLP, pages 41–48.
  16. Idan Szpektor, Eyal Shnarch, and Ido Dagan. 2007. Instance-based evaluation of entailment rule acquisition. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 456–463, Prague, Czech Republic, June. Association for Computational Linguistics. find
  17. Idan Szpektor, Ido Dagan, Roy Bar-Haim, and Jacob Goldberger. 2008. Contextual preferences. In Proceedings of ACL-08: HLT, pages 683–691, Columbus, Ohio, June. Association for Computational Linguistics. find
  18. Rui Wang and Günter Neumann. 2007. Recognizing textual entailment using sentence similarity based on dependency tree skeletons. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 36–41, Prague, June. Association for Computational Linguistics.
  19. Fabio Massimo Zanzotto and Alessandro Moschitti. 2006. Automatic learning of textual entailments with cross-pair similarities. In ACL-44: Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 401–408, Morristown, NJ, USA. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dinu-2009-inference-rules-their-application-recognizing/