📄 Recognizing Textual Entailment using Dependency Analysis and Machine Learning

Summary

The paper treats recognizing textual entailment (RTE) as binary classification. Given a text T and a hypothesis H, the system decides whether H follows from T. The authors object to earlier ML-based RTE systems that use many opaque features, and they build a classifier on only 8 simple, interpretable features that are computed automatically.

Each sentence is parsed with the Stanford Dependency parser. The dependency relations are then turned into a slot-filler frame with these fields: subject, subject modifier, object, object modifier, verb, verb complements, negation and number. Most features compare the T frame with the H frame. Lexical knowledge comes from WordNet (verb root forms) and VerbOcean (antonymy and similarity between verbs). Naïve Bayes, SVM and Decision Tree classifiers were trained in Weka on RTE1, RTE2 and RTE3.

On the test sets, Decision Trees did best, reaching roughly 60–63% accuracy. That is competitive with ML systems that use two to three times as many features. The work suggests that a few well-motivated syntactic checks can replace large, hard-to-explain feature sets.

Key points

• The feature set has 8 features: one mismatch feature (negation check) and seven match features. These include:
  • word overlap,
  • number agreement,
  • dependency overlap,
  • syntactic role match on (subject, object, verb) tuples,
  • a check that links verbs to their noun forms.
• Test accuracy with DT was 60.12 (RTE1), 59.87 (RTE2) and 62.75 (RTE3). SVM with an RBF kernel was close behind, and Naïve Bayes was weakest.
• Adding the RTE task label as an extra feature raised DT accuracy to 61.25 / 60.41 / 64.38. This repeats an observation by Bos and Markert (2005).
• The system beats Inkpen et al., Kozareva & Montoyo, Pakray et al., MacCartney et al. and Bos & Markert. It does not beat Hickl et al. (65.25 on RTE2, 12 features) or Adams et al. (67 on RTE3, 13 features).
• In the per-task breakdown on RTE1, the system outperforms Pazienza et al. and Bos & Markert. The Comparable Documents subset is the easiest, with accuracy above 80%.

Interesting ideas

• I1. Dependency overlap counts shared governor–dependent word pairs and ignores the relation label, so an active sentence and its passive paraphrase still match strongly (5 of 6 pairs in the paper's example).
• I2. The parser output is flattened into a small slot-filler record (subject, object, verb, their modifiers, negation, numbers), and most comparisons run on these slots rather than on raw words.
• I3. Negation is treated as either explicit (a neg dependency) or implicit, where H's verb is an antonym of T's verb (rise vs. drop), checked on lemmas through VerbOcean.
• I4. Number agreement compares numeric modifiers attached to the same governing noun, so a price stated in T and a different one in H count as a contradiction.
• I5. One feature links a verb to its noun form in the other sentence (died ↔ death, bled ↔ blood), catching paraphrases that move content between verb and noun.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): a cheap first-pass score for whether a candidate answer sentence matches a question's tree, robust to voice changes, before full frame comparison.
• I2 (frames): a minimal, explainable fallback frame that prag can emit when VerbVerbNet role assignment fails, so downstream comparison still has who/what/did slots.
• I3 (world): when merging statements into the text's world, flag a state conflict if two events on the same entity carry opposite-polarity verbs, not only literal "not".
• I4 (world): attach quantities to their head entity in the world model so that answer checks can detect mismatched amounts tied to the same thing.
• I5 (frames): use verb–nominalization links from the compiled lexicon so an event mentioned as a noun can still fill the same frame as its verbal version.