📄 Combining Lexical-Syntactic Information with Machine Learning for Recognizing Textual Entailment

Summary

This paper describes the SINAI group's (Universidad de Jaén) first entry in the PASCAL textual entailment challenge. The task is to decide whether a hypothesis follows from a given text. The authors treat it as binary classification over hand-designed features, with two feature families. The lexical family measures word overlap between hypothesis and text after stopword removal and GATE stemming: soft matching via WordNet-based Lin similarity, exact matching, consecutive-subsequence matching and trigram matching. The syntactic family parses both sides with the COLLINS parser and finds terms shared by the two trees. It then compares how far apart those terms sit in each tree, both as path length in nodes and as depth, and summarises the differences with simple statistics. Two classifiers, BBR (Bayesian logistic regression) and TiMBL, were trained on the challenge development data. The best accuracy was close to 65%, from BBR with lexical and syntactic features together. Lexical features alone came within about one point of that, which suggests shallow overlap already captures much of the signal.

Key points

• Lexical features: Simple Matching (Lin similarity over WordNet, threshold 0.5, normalised by hypothesis length), Binary Matching (exact stem equality), Consecutive Subsequence Matching (n-grams of length 2 up to the hypothesis length, normalised per length) and Trigrams.
• Syntactic features come from COLLINS parses. They include the count of aligned terms, matched and unmatched POS labels of aligned terms, and the average, minimum, maximum and standard deviation of path-distance differences and depth differences between aligned term pairs.
• Results by classifier:
  • BBR: 0.6475 (3 lexical + syntactic), 0.64625 (4 lexical + syntactic) and 0.63875 (3 lexical only).
  • TiMBL: 0.6062, 0.6037 and 0.57.
• The official submission was Exp5 (TiMBL, three lexical features), which scored 0.6037.
• Adding syntax improved on lexical-only by less than 2% in the authors' words. They conclude that semantic features deserve more attention than syntactic ones.
• Planned future work: a voting ensemble of TiMBL, SVM and BBR, and a hierarchical architecture built on constraint satisfaction networks.

Interesting ideas

• I1. For every pair of words shared by two sentences, compare their tree distance in one parse with their distance in the other; the spread of these differences shows whether the shared words keep the same structural relations, and each pair is easy to inspect.
• I2. The paper keeps exact matching and WordNet-based soft matching (Lin similarity with a threshold) as separate signals instead of merging them, so it stays visible whether a match was literal or only near-synonymous.
• I3. The ablation shows that plain word overlap gets within a point or two of the full lexical-plus-syntax system, so a cheap lexical baseline works as a test of whether structure adds anything.
• I4. When the same aligned term gets different POS labels in text and hypothesis, the paper turns that disagreement into a feature, a cheap marker of a possible mismatch or tagging error.
• I5. Consecutive-subsequence matching gives credit for runs of every length from 2 upward, normalised per length, so it rewards preserved word order more finely than a single n-gram size.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): a reader could check whether a question or claim is supported by a text by aligning lemmas and comparing UD paths between them, then report the specific pairs whose paths diverge as the explanation.
• I2 (world): when linking hypothesis entities to entities in the text world, tag each link as exact-lemma or ontology-similar, so the answer can state which kind of evidence it rests on.
• I3 (learn): run a bag-of-words overlap scorer as a negative control on every entailment or QA evaluation, and treat tree-based gains that do not clearly beat it as unproven.
• I4 (tokens): flag lemmas that the tagger labels inconsistently across closely related sentences as candidates for tagger errors or for review when building silver data.
• I5 (back): use it as a graded faithfulness score when comparing original text with the output of the text → graph → text compressor, to see at what span length ordering starts to break.