📄 Textual Entailment Through Extended Lexical Overlap and Lexico-Semantic Matching

Summary

The paper addresses Recognizing Textual Entailment: given a text T and a hypothesis H, decide whether H follows from T. It was evaluated in the PASCAL RTE3 challenge, which has 800-pair development and test corpora drawn from QA, IR, IE and SUM outputs. The authors compare two classifiers, both trained as decision trees.

• Extended lexical overlap (ELO): links each hypothesis token to its best-matching text token. Matches are scored with WordNet paths plus web co-occurrence counts. Features are then derived from this "token map".
• Lexico-semantic matching (LSM): aligns chunks through signed WordNet and coreference relations. It also compares edges in dependency graphs enriched with PropBank arguments.

Surprisingly, the shallower ELO beat LSM in almost every cross-validation setting. ELO reached 67% on RTE3 test, up from its 62.8% at RTE2. The authors attribute LSM's weakness to errors that pile up from its imperfect parser, coreference resolver and semantic labeler. The work is a useful cautionary result about pipeline depth versus tool reliability.

Key points

• ELO similarity. A WordNet path score (Hirst & St-Onge style, with Up/Down/Horizontal relation groups and penalties for changing direction) is combined with an AltaVista-based lexical entailment probability (after Glickman et al.). The combination is non-linear, with α = 0.1.
• ELO features. The product of all mapping scores, a score flipped by the parity of unmapped negations, and the task domain. These feed Weka J48.
• LSM design.
  • Hand-tuned relation scores: identity 1.0, coreference/synonymy 0.8, antonymy −0.8, hyponymy −0.5, holonymy −0.4.
  • Dependency graphs come from the de Marneffe et al. parser (48 relations).
  • Multi-sentence graphs are merged through coreferent, identical or nn-linked nodes.
• Cross-validation (10-fold). On RTE3Dev, ELO scored 66.38 and LSM 63.63. ELO+LSM with JRip scored 67.50, the only case where the combination won. Accuracy fell as RTE1/RTE2 data were pooled in, down to about 60 for ELO.
• RTE3 test, ELO by task. QA 80.0, IR 73.5, SUM 61.0, IE 53.5, overall 67.0.

Interesting ideas

• I1. Relation links carry a direction-aware sign: moving to a more general term (hypernym) supports entailment, while moving to a more specific term (hyponym) or an antonym counts against it. This turns lexical knowledge into evidence for and against rather than plain similarity.
• I2. Negations are handled by parity: an odd count of uncovered negators inverts the score, and a negated item aligned via antonymy cancels to positive.
• I3. A secondary evidence source only fills the "slack" left by the primary one: the weaker score is weighted by one minus the stronger score, so it matters mainly when the main score is low.
• I4. Using a product of per-token scores makes the weakest link cap the whole judgment, so a single unsupported hypothesis element sinks the entailment.
• I5. Error analysis showed that pooling heterogeneous datasets lowers scores but gives a more honest estimate. It also showed that deep pipelines can lose to shallow ones when their components are noisy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): Ontology lookups from WordNet could return signed, directional compatibility, so the text world can check whether a hypothesis entity is licensed by a text entity.
• I2 (frames): Polarity on frames could be composed multiplicatively along the chain of negators, so double negation and negated antonyms resolve explicitly and explainably.
• I3 (learn): When combining witness parsers or teacher tiers, a trusted source could dominate and weaker witnesses could contribute only where it is uncertain.
• I4 (answer): A reader verifying a candidate answer could require every hypothesis frame element to be grounded, and report the failing element as the explanation.
• I5 (learn): Evaluate on mixed-source pools, and ablate each upstream component's error contribution so a deeper stage is not credited or blamed wrongly.