Elamur Research⌕
☰
home / papers / 2009 / From Lexical Entailment to Recognizing Textual Entailment Using Linguistic Resources

From Lexical Entailment to Recognizing Textual Entailment Using Linguistic Resources

Paper · 2009 · Bahadorreza Ofoghi and John Yearwood

Summary

The paper describes a two-way Recognizing Textual Entailment (RTE) system that the authors entered in TAC 2009. It decides whether a text T supports a hypothesis H, or whether the pair is No Entailment. The approach is term-based but adds a syntactic step first. Sentences are lightly cleaned, split with LingPipe and parsed with the Link Grammar Parser (LGP). Atomic propositions are then pulled out using seven existing linkage rules plus three new ones, and words are lemmatized with TreeTagger. H is accepted only if each of its propositions has a counterpart in T whose words cover all of the proposition's words. A word matches another if they are identical, synonyms or near hypernyms in WordNet, or (for verbs) members of the same FrameNet frame. When no propositions can be extracted, a character-level Levenshtein threshold is used instead. Accuracy stays at about 0.49–0.50 on the RTE4 and RTE5 data, and entailment recall is very low. A failure analysis traces most errors to parsing and proposition extraction rather than to the lexical resources. This is a useful warning for any pipeline that depends on its parser.

Key points

  • Evaluation data:
    • TAC-RTE 2008 test (rte4, 1000 pairs).
    • TAC-RTE 2009 main-task dev (rte5, 600 pairs).
    • TAC-RTE 2009 main-task test (rte5, 600 pairs).
  • Baseline configuration:
    • Verbs are expanded through FrameNet and nouns through WordNet.
    • The hypernym distance is set to 1.
    • The Levenshtein fallback threshold is 3.
    • Every hypothesis term must find a match.
  • Macro-averaged accuracy is 0.496 on rte4-test, 0.490 on rte5-dev and 0.500 on rte5-test. The TAC 2009 field ranged from 0.5000 (low) through 0.6117 (median) to 0.7350 (high).
  • The system is heavily biased toward No Entailment:
    • Entailment recall is 0.140 on rte4 (70 of 500) and 0.083 on rte5-dev (25 of 300).
    • No Entailment recall is 0.852 and 0.896.
  • On rte4, LGP returned no parse for 394 hypotheses and texts, so only about 60% were parsed. Proposition-level matching was possible for only about 23% of rte4 pairs and about 29% of rte5-dev pairs.

Interesting ideas

  • I1. Break H into atomic propositions and require each one to be supported by some proposition of T; one unsupported proposition blocks entailment.
  • I2. Treat two verbs as related when they share a FrameNet frame, which gives event-level similarity that WordNet's synonym and hypernym links do not capture.
  • I3. Cap hypernym climbing at a small number of links so that matching does not drift into overly general concepts.
  • I4. Attribute failures to pipeline stages by counting items with no parse and items with no extracted units, rather than only reporting end accuracy.
  • I5. Report per-class recall next to accuracy to expose a system that mostly predicts one class and so scores near the lowest result in the TAC field.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Readers could check every frame of a question or claim separately against the text world, which makes it explicit which sub-claim was not supported.
  • I2 (world): Frame co-membership could become a separate, labelled relation in the compiled ontology, so that matching predicates via a shared frame stays traceable.
  • I3 (world): Mova could expose hypernym depth as an explicit, logged parameter whenever a text entity is matched against a query concept.
  • I4 (learn): Evaluation reports could track coverage at the tree and frames stages, so that answer-level errors are separated from upstream extraction losses.
  • I5 (learn): Mova could add trivial-baseline controls, such as always answering "no", to its test suites so that apparent accuracy is checked against degenerate behaviour.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Federation University.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Bahadorreza Ofoghi and John Yearwood. 2009. From Lexical Entailment to Recognizing Textual Entailment Using Linguistic Resources. Proceedings of the Australasian Language Technology Association Workshop 2009.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ofoghi2009lexical,
  title = {{From Lexical Entailment to Recognizing Textual Entailment Using Linguistic Resources}},
  author = {Bahadorreza Ofoghi and John Yearwood},
  booktitle = {Proceedings of the Australasian Language Technology Association Workshop 2009},
  year = {2009},
  url = {https://aclanthology.org/U09-1018/}
}

References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Elena Akhmatova and Diego Molla. 2006. Recognizing textual entailment via atomic propositions. In Proceedings of the Machine Learning Challenges Workshop (MLCW), 385–403. Southampton, UK. find
  2. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of the 17th International Conference on Computational Linguistics (COLING, 86–90. Universite de Montreal, Montreal, Quebec, Canada.
  3. W.E. Bosma and C. Callison-Burch. 2006. Paraphrase substitution for recognizing textual entailment. In Working Notes of CLEF 2006, 1–8. Alicante, Spain. find
  4. Peter Clark and Phil Harrison. 2008. Recognizing textual entailment with logic inference. In Proceedings of the Fourth PASCAL Challenges Workshop on Recognizing Textual Entailment. Gaithersburg, Maryland, USA. find
  5. M. Kouylekov and B. Magnini. 2005. Recognizing textual entailment with tree edit distance algorithms. In Proceedings of the First PASCAL Challenges Workshop on Recognizing Textual Entailment, 17–20. Southampton, UK. find
  6. D. Lin and P. Pantel. 2001. DIRT - Discovery of inference rules from text. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 323–328. San Francisco, California, USA. find
  7. George A. Miller, R. Beckwith, C. Fellbaum, D. Gross, and K. J. Miller. 1990. Introduction to WordNet: An on-line lexical database. International Journal of Lexicography, 3(4):235–244. ↗
  8. M. T. Pazienza, M. Pennacchiotti, and F. M. Zanzotto. 2005. Textual entailment as syntactic graph distance: A rule based and a SVM based approach. In Proceedings of the First PASCAL Challenges Workshop on Recognizing Textual Entailment, 25–28. Southampton, UK. find
  9. Alvaro Rodrigo, Anselmo Penas, and Felisa Verdejo. 2008. Towards an entity-based recognition of textual entailment. In Proceedings of the Fourth PASCAL Challenges Workshop on Recognizing Textual Entailment. Gaithersburg, Maryland, USA. find
  10. R. de Salvo Braz, R. Girju, V. Punyakanok, D. Roth, and M. Sammons. 2005. Textual entailment recognition based on dependency analysis and WordNet. In Proceedings of the First PASCAL Challenges Workshop on Recognizing Textual Entailment, 29–32. Southampton, UK. find
  11. Daniel Sleator and Davy Temperley. 1993. Parsing English with a link grammar. In Proceedings of the Third International Workshop on Parsing Technologies.
  12. Helmut Schmid. 1994. Probabilistic part-of-speech tagging using decision trees. In Proceedings of the Conference on New Methods in Language Processing. Manchester, UK. find
  13. Marta Tatu and Dan Moldovan. 2005. A semantic approach to recognizing textual entailment. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing (HLT-EMNLP), 371–378. Vancouver, British Columbia, Canada.
  14. Mehmet Ali Yatbaz. 2008. RTE4: Normalized dependency tree alignment using unsupervised n-gram word similarity score. In Proceedings of the Fourth PASCAL Challenges Workshop on Recognizing Textual Entailment. Gaithersburg, Maryland, USA. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ofoghi-2009-lexical-entailment-recognizing-textual-entailment/