Elamur Research⌕
☰
home / papers / 2007 / DLSITE-2: Semantic Similarity Based on Syntactic Dependency Trees Applied to Textual Entailment

DLSITE-2: Semantic Similarity Based on Syntactic Dependency Trees Applied to Textual Entailment

Paper · 2007 · Daniel Micol, Óscar Ferrández, Rafael Muñoz et al.

Summary

DLSITE-2 is a system from the University of Alicante that decides whether a hypothesis follows from a text (textual entailment). It relies almost entirely on syntax and is built to run fast enough for interactive use. Both sentences are parsed with MINIPAR, and the dependency trees are stripped of grammatically unimportant tokens. The system then checks whether the hypothesis tree fits inside the text tree. This check is relaxed in two ways: extra intermediate nodes are allowed, and words may match if their WordNet Wu-Palmer similarity is high enough. If that strict test fails, a fallback compares the two trees by identical lemmas. Each matched node counts more when it has an important category or relation and when it sits close to the root, and the total is normalised by the hypothesis weight. On the RTE-2 test set the system reaches 60.75% accuracy at roughly 1.3 seconds per pair. This beats the group's own earlier RTE-2 entry. It shows that a light, explainable tree-based method can be competitive.

Key points

  • Evaluation used RTE-2 development and test sets of 800 pairs each. Overall test accuracy was 60.75% and average precision 57.91%. By task, SUM was best (68.50%) and IE was weakest (50.50%).
  • Two tokens count as a match in the embedding test when their Wu-Palmer similarity (WordNet::Similarity) reaches at least 80%. The setting was chosen empirically.
  • The node weight multiplies three factors: a category weight, a relation weight, and a depth decay of µ^(−depth) with µ = 1.1.
    • Category weights: content verbs 1.0, nouns/numbers 0.75, copular Be 0.7, adjectives/adverbs/noun-noun modifiers 0.5, Have/Be 0.3.
    • Relation weights: subjects/objects 1.0, all other relations 0.5.
  • The embedding test is very strict. Only 3 development pairs and 1 test pair passed it, so most decisions come from the weighted lemma alignment.
  • Runtime was about 1.28–1.31 s per pair on a 2 GHz Core 2 Duo. The previous RTE-2 system scored 55.63%, so the new one is roughly 9.2% better in relative terms.

Interesting ideas

  • I1. The tree is pruned by a whitelist of grammatical categories that was tuned through ablation: each category was tested both kept and dropped. This removes determiners, auxiliaries and complementizers while keeping the tree's shape, which reduces both noise and runtime.
  • I2. Tree inclusion is checked top-down while allowing "gaps": a hypothesis branch matches if its nodes appear in the same order along a text branch, even with extra nodes in between. This tolerates modifiers and inserted material without giving up structure.
  • I3. Similarity is weighted by node importance: category weight times relation weight times a decay with depth, normalised by the total weight of the hypothesis. A missing main verb or subject therefore costs far more than a missing adverb.
  • I4. The authors' error analysis shows that appositions break verb-rooted matching. In "Tony Blair, the British Prime Minister, met…", the hypothesis's "is" has no counterpart in the text. They propose rewriting "X, Y, …" as "X is Y, and …" before matching.
  • I5. The system outputs a continuous score, and the entailment threshold is chosen on a development corpus rather than hard-coded.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Before matching frames between a question and a text, Mova could drop function-word nodes from the UD tree using a list validated by ablation, keeping only content-bearing heads.
  • I2 (answer): A reader judging whether a statement follows from a fable could test gapped subtree inclusion over UD trees, which gives an explainable yes together with the exact matched path.
  • I3 (answer): Partial-credit scoring for school-test answers could weight missing predicates and core arguments above peripheral modifiers, and each penalty would be traceable to a specific node.
  • I4 (world): When building the world of a text, appositions (UD appos) should yield explicit identity or attribute statements, so that implicit "is" facts can be queried like stated ones.
  • I5 (learn): Mova's decision thresholds could be calibrated on held-out development sets, and each calibrated threshold should be checked against negative controls rather than fixed by hand.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AlicanteAlicante · Spain21 works · 1,098 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Daniel Micol, Óscar Ferrández, Rafael Muñoz and Manuel Palomar. 2007. DLSITE-2: Semantic Similarity Based on Syntactic Dependency Trees Applied to Textual Entailment. Proceedings of the Second Workshop on TextGraphs: Graph-Based Algorithms for Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{micol2007dlsite,
  title = {{DLSITE-2: Semantic Similarity Based on Syntactic Dependency Trees Applied to Textual Entailment}},
  author = {Daniel Micol, Óscar Ferrández, Rafael Muñoz and Manuel Palomar},
  booktitle = {Proceedings of the Second Workshop on TextGraphs: Graph-Based Algorithms for Natural Language Processing},
  year = {2007},
  url = {https://aclanthology.org/W07-0211/}
}

References

15 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rod Adams. 2006. Textual Entailment Through Extended Lexical Overlap. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  2. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The Second PASCAL Recognising Textual Entailment Challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  3. Johan Bos, and Katja Markert. 2006. When logical inference helps determining textual entailment (and when it doesnt). In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  4. Óscar Ferrández, Rafael M. Terol, Rafael Muñoz, Patricio Martı́nez-Barco, and Manuel Palomar. 2006. An approach based on Logic Forms and WordNet relationships to Textual Entailment performance. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  5. Jesús Herrera, Anselmo Peñas, Álvaro Rodrigo, and Felisa Verdejo. 2006. UNED at PASCAL RTE-2 Challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  6. Andrew Hickl, John Williams, Jeremy Bensley, Kirk Roberts, Bryan Rink, and Ying Shi. 2006. Recognizing Textual Entailment with LCC’s GROUNDHOG System. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  7. Sophia Katrenko, and Pieter Adriaans. 2006. Using Maximal Embedded Syntactic Subtrees for Textual Entailment Recognition. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  8. Dekang Lin. 1998. Dependency-based Evaluation of MINIPAR. In Workshop on the Evaluation of Parsing Systems, Granada, Spain. find
  9. Bill MacCartney, Trond Grenager, Marie-Catherine de Marneffe, Daniel Cer, and Christopher D. Manning. 2006. Learning to recognize features of valid textual entailments. In Proceedings of the North American Association of Computational Linguistics (NAACL-06), New York City, New York, United States of America. find
  10. George A. Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J. Miller. 1990. Introduction to WordNet: An On-line Lexical Database. International Journal of Lexicography 1990 3(4):235-244. ↗
  11. Ted Pedersen, Siddhart Patwardhan, and Jason Michelizzi. 2004. WordNet::Similarity - Measuring the Relatedness of Concepts. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL-04), Boston, Massachussets, United States of America. find
  12. Rion Snow, Lucy Vanderwende, and Arul Menezes. 2006. Effectively using syntax for recognizing false entailment. In Proceedings of the North American Association of Computational Linguistics (NAACL-06), New York City, New York, United States of America. find
  13. Marta Tatu, Brandon Iles, John Slavick, Adrian Novischi, and Dan Moldovan. 2006. COGEX at the Second Recognizing Textual Entailment Challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find
  14. Zhibiao Wu, and Martha Palmer. 1994. Verb Semantics and Lexical Selection. In Proceedings of the 32nd Annual Meeting of the Associations for Computational Linguistics, pages 133-138, Las Cruces, New Mexico, United States of America. ↗
  15. Fabio M. Zanzotto, Alessandro Moschitti, Marco Pennacchiotti, and Maria T. Pazienza. 2006. Learning textual entailment from examples. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Venice, Italy. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/micol-2007-dlsite-2-semantic-similarity-based/