📄 Recognizing Textual Entailment Using Sentence Similarity based on Dependency Tree Skeletons

Summary

The paper addresses Recognizing Textual Entailment (RTE): deciding whether a text T supports a hypothesis H. Three observations about the RTE data guide the approach. H is usually shorter than T, much of T is irrelevant to the decision, and what matters most is how the relations between shared topics differ. The system therefore works backwards from H. It anchors on nouns that appear in both sentences and cuts a compact "tree skeleton" out of each dependency parse, keeping only the paths from those nouns up to a root. Each skeleton is then generalized into a sequence of coarse relation tags. The differences between the two sequences, plus checks on the root verb and its argument relations, form an Entailment Pattern, which a subsequence kernel classifies. Pairs the patterns cannot handle fall back to a triple-overlap score and a bag-of-words score. The system uses no WordNet, hand-written rules or extra corpora, yet reaches 71.1% (10-fold cross-validation, RTE-3 dev) and 66.9% (RTE-3 test). That score would have ranked in the top three at RTE-2.

Key points

• Results:
  • 71.1% accuracy with Minipar + kernel + backups (10-fold CV on the 800-pair RTE-3 dev set).
  • 70.1% with the Stanford Parser in the same setup.
  • 66.9% on the RTE-3 test set when combining both parsers.
• Coverage of the skeleton method:
  • Skeletons could be built for only 254 dev pairs, about 32% of the data.
  • On the pairs it covers, accuracy is roughly 74.8% (CV) and 68.8% (test).
  • QA pairs covered by patterns score about 91%.
• The method shines on IE pairs, which are mostly binary relations: 74% versus 54.5% for the BoW baseline. BoW stays best on SUM pairs.
• An Entailment Pattern is a quadruple <LSD, RSD, VC, VRC>:
  • left and right spine differences (LSD, RSD);
  • verb consistence (VC);
  • a three-valued verb-relation consistence (VRC).
• Classification uses SMO from WEKA. Extra training pairs built from MUC, TREC and news did not raise overall accuracy, because they only helped pairs that were already covered.

Interesting ideas

• I1. Use the hypothesis to decide which part of the text to examine: find shared nouns, then keep only the dependency paths linking them in T, which discards distracting material before any comparison.
• I2. Turn a two-spine skeleton into a linear sequence (left spine, root, right spine), strip the shared prefix and suffix, and treat the leftover middle as the signal, so that "what differs" becomes an explicit, inspectable object.
• I3. Flag a hypothesis whose subject and object have swapped relative to the text, even when the same words and verb appear, as a separate feature from verb-meaning similarity.
• I4. Match dependency triples approximately at several relaxation levels (ignore head, ignore dependent, or ignore the label) and weight each level differently, using weights learned from data and normalized by hypothesis size.
• I5. Report coverage and accuracy separately per task type, and route uncovered cases to simpler fallbacks. This showed that adding data only improved cases the method already handled.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): A reader answering a question could locate the question's anchor entities in the text's tree and compare only the connecting subgraph, keeping each judgment small and explainable.
• I2 (world): When checking whether a claim matches a stored relation, the leftover path difference could be logged as the reason for a mismatch rather than reported as a bare score.
• I3 (frames): Frame comparison could carry an explicit role-consistency check (agent vs patient swapped), catching "X bought Y" vs "Y bought X" errors without lexical knowledge.
• I4 (answer): A graded triple match with labelled relaxation levels would give partial-credit answers that still say exactly which part of the relation was missing.
• I5 (learn): Evaluations should break down "how often the structured path fires" versus "how accurate it is when it fires," so that extra silver data is not mistaken for progress on coverage.