Elamur Research⌕
☰
home / papers / 2010 / Syntactic/Semantic Structures for Textual Entailment Recognition

Syntactic/Semantic Structures for Textual Entailment Recognition

Paper · 2010 · Yashar Mehdad, Alessandro Moschitti and Fabio Massimo Zanzotto

Summary

The paper addresses Recognizing Textual Entailment (RTE): deciding whether a hypothesis follows from a text. Many systems rely on lexical-syntactic rewrite rules. Rules mined without supervision from corpora are noisy and sparse, and semantic rules are usually written by hand. The authors propose a supervised alternative built from off-the-shelf parsers and lexical resources. Text and hypothesis are represented by constituency parse trees and compared with a syntactic/semantic tree kernel (SSTK). This kernel lets two tree fragments match when their leaf words are semantically related, not only when they are identical. Word relatedness comes from two sources:

  • WordNet path-based measures (Path, WUP);
  • LSA, computed over Wikipedia or the BNC.

Within a kernel-based RTE system, these kernels are combined with a word-overlap kernel and placeholder-based rule kernels. Experiments on RTE2, RTE3 and RTE5 show that Wikipedia-based LSA consistently beats the purely syntactic baseline of Zanzotto and Moschitti, with statistically significant gains. The system ranks near the top of each challenge without task-specific tuning. LSA also turns out to be cheaper and broader in coverage than WordNet.

Key points

  • Lexical similarity is plugged into the tree kernel of Collins and Duffy (STK), giving SSTK; "max" variants (maxSTK, maxSSTK) model rules with variables. The gains over the no-semantics kernel are significant (p << 0.05).
  • The word-overlap kernel WOK alone reaches 60.62 (RTE2), 66.75 (RTE3) and 60.67 (RTE5), and removing it always hurts badly.
  • maxSSTK+WOK improves on WOK on all three datasets. STK and maxSTK fail to help on RTE3.
  • Coverage of dataset content words is about 0.83 for Wikipedia, versus 0.41–0.55 for BNC and 0.34–0.42 for WordNet. One similarity lookup costs 0.54 ms with LSA, compared with 5.3 ms (WordNet with POS) and 15.2 ms (WordNet without POS).
  • The system ranks 3rd of 23 (RTE2), 4th of 26 (RTE3) and 4th of 20 (RTE5). On RTE2 it beats the best FrameNet-based system, which scored 62.5.

Interesting ideas

  • I1. Soft structural matching: tree fragments count as shared when their words are related by a similarity score, so "killed Kennedy" can align with "murdered JFK" without a hand-written rule.
  • I2. Generalizing rules through lexical relatedness: a rule learned from one example should transfer only to cases whose words are semantically close. This stops a pattern like "X verb Y → Y died" from firing on "met".
  • I3. Choose the knowledge source by coverage and speed, not only by accuracy. Wikipedia LSA covered about twice as many words and was roughly 10× faster than WordNet, at comparable accuracy.
  • I4. Ablate each component, including the simple lexical-overlap baseline. The overlap kernel proved essential, and syntax helped on some datasets but not on RTE3.
  • I5. Tune only one interpretable knob, the positive/negative cost ratio. Because semantics tends to raise recall, they lowered j slightly to recover precision.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Our tree comparisons could match UD subtrees whose lemmas are near-synonyms, using a graded match while keeping explicit, inspectable alignments.
  • I2 (frames): When frames induce role-mapping rules (killer/victim → protagonist of DEATH), they could gate transfer to new verbs by VerbNet class or similarity, which blocks spurious inferences.
  • I3 (world): Our compiled knowledge layer could measure per-resource coverage on real test texts and fall back to a broad distributional table when WordNet/FrameNet has no entry.
  • I4 (learn): Evaluations should always report a shallow-overlap control alongside the structured pipeline, to show where the tree and frame steps actually add value.
  • I5 (answer): Entailment-style readers could expose a single precision/recall bias parameter, set once on validation data rather than per test set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TrentoTrento · Italy62 works · 14,868 citations🎓University of Rome Tor VergataRome · Italy6 works · 1,251 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 510 works

Cite

Cite as Yashar Mehdad, Alessandro Moschitti and Fabio Massimo Zanzotto. 2010. Syntactic/Semantic Structures for Textual Entailment Recognition. Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mehdad2010syntactic,
  title = {{Syntactic/Semantic Structures for Textual Entailment Recognition}},
  author = {Yashar Mehdad, Alessandro Moschitti and Fabio Massimo Zanzotto},
  booktitle = {Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/N10-1146/}
}

References

36 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Agirre, E. Alfonseca, K. Hall, J. Kravalova, M. Paşca, and A. Soroa. 2009. A study on similarity and relatedness using distributional and wordnet-based approaches. In NAACL ’09: Proc. HLT/NAACL.
  2. R. Bar-Haim, I. Dagan, B. Dolan, L. Ferro, D. Giampiccolo, and I. Magnini, B. Szpektor. 2006. The ii PASCAL recognising textual entailment challenge. In Proc. of the II PASCAL Challenges Workshop. find
  3. R. Bar-Haim, I. Dagan, I. Greental, and E. Shnarch. 2007. Semantic inference at the lexical-syntactic level. In AAAI’07: Proc. of the 22nd national conference on Artificial intelligence. find
  4. R. Bar-Haim, J. Berant, and I. Dagan. 2009. A compact forest for scalable inference over entailment and paraphrase rules. In Proc. of EMNLP. find
  5. R. Basili, M. Cammisa, and A. Moschitti. 2005. Effective use of wordnet semantics via kernel-based learning. In CoNLL. find
  6. R. Basili, M. Cammisa, and A. Moschitti. 2006. A semantic kernel to classify texts with very few training examples. In Informatica. find
  7. S. Bloehdorn and A. Moschitti. 2007a. Combined syntactic and semantic kernels for text classification. In ECIR. find
  8. S. Bloehdorn and A. Moschitti. 2007b. Structure and semantics for expressive text kernels. In Proc. of CIKM ’07. find
  9. S. Bloehdorn, R. Basili, M. Cammisa, and A. Moschitti. 2006. Semantic kernels for text classification based on topological measures of feature similarity. In Proc. of ICDM 06, Hong Kong, 2006. find
  10. J. Bos and K. Markert. 2005. Recognising textual entailment with logical inference. In HLT ’05: Proc. of the conference on HLT and EMNLP.
  11. A. Burchardt, N. Reiter, S. Thater, and A. Frank. 2007. Semantic Approach to Textual Entailment: System Evaluation and Task Analysis. In Proc. of the 3rd-PASCAL Workshop on Textual Entailment, Prague. find
  12. Y. S. Chan and H. T. Ng. 2005. Word sense disambiguation with distribution estimation. In Proc. of IJCAI’05. find
  13. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proc. of the 1st NAACL conference.
  14. M. Collins and N. Duffy. 2002. New ranking algorithms for parsing and tagging: kernels over discrete structures, and the voted perceptron. In Proc. of ACL ’02. find
  15. N. Cristianini and R. Holloway. 2001. Latent semantic kernels. ↗
  16. G. Dinu and R. Wang. 2009. Inference rules and their application to recognizing textual entailment. In Proc. of the EACL ’09. find
  17. D. Giampiccolo, B. Magnini, Ido Dagan, and B. Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proc. of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. find
  18. Claudio Giuliano. 2007. jLSI a for latent semantic indexing. http://tcc.itc.it/research/textec/tools-resources/jLSI.html. ↗
  19. A. D. Haghighi, A. Y. Ng, and C. D. Manning. 2005. Robust textual inference via graph matching. In HLT ’05: Proc. of the conference on HLT and EMNLP. find
  20. David Haussler. 1999. Convolution kernels on discrete structures. Technical report. find
  21. J. J. Jiang and D. W. Conrath. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proc. of the 10th ROCLING. ↗
  22. Thorsten Joachims. 1999. Making large-scale support vector machine learning practical. ↗
  23. Landauer, Foltz, and Laham. 1998. Introduction to latent semantic analysis. In Discourse Processes 25. find
  24. D. Lin and P. Pantel. 2001. DIRT-discovery of inference rules from text. In Proc. of the ACM KDD-01. find
  25. R. Mihalcea, C. Corley, and C. Strapparava. 2006. Corpus-based and knowledge-based measures of text semantic similarity. In Proc. of AAAI06. find
  26. Alessandro Moschitti. 2006. Making tree kernels practical for natural language learning. In Proc. of EACL. find
  27. Peter Norvig. 1987. A unified theory of inference for text understanding. Technical report, USA. find
  28. Sebastian Padó, 2006. User’s guide to sigf: Significance testing by approximate randomisation. find
  29. T. Pedersen, S. Patwardhan, and J. Michelizzi. 2004. Wordnet::similarity - measuring the relatedness of concepts. In Proc. of 5th NAACL. find
  30. D. Roth and M. Sammons. 2007. Semantic and logical inference model for textual entailment. In Proc. of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. find
  31. I. Szpektor and I. Dagan. 2008. Learning entailment rules for unary templates. In Proc. of COLING ’08. find
  32. M. Tatu and D. Moldovan. 2005. A semantic approach to recognizing textual entailment. In HLT ’05: Proc. of HLT/EMNLP. find
  33. Z. Wu and M. Palmer. 1994. Verb semantics and lexical selection. In Proc. of ACL. ↗
  34. Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In Proc. of ACL 2000, Morristown, NJ, USA. ↗
  35. F. M. Zanzotto and A. Moschitti. 2006. Automatic learning of textual entailments with cross-pair similarities. In Proc. of ACL ’06. find
  36. F. M. Zanzotto, M. Pennacchiotti, and A. Moschitti. 2009. A machine learning approach to textual entailment recognition. NATURAL LANGUAGE ENGI-NEERING. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mehdad-2010-syntactic-semantic-structures-textual-entailment/