📄 A Study of Machine Learning Algorithms for Recognizing Textual Entailment

Summary

The paper addresses Recognizing Textual Entailment (RTE): deciding whether a hypothesis H follows from a text T. It covers the classic two-way setting and the three-way setting (entailment / contradiction / unknown) used in RTE4. The author fixes a small set of four similarity features and compares five Weka classifiers on equal terms: SVM, AdaBoost, BayesNet, Multilayer Perceptron and Decision Trees. The features are:

• character-level Levenshtein distance;
• a token-level lexical distance;
• a WordNet-based semantic similarity;
• a longest common substring score.

The classifiers are trained on every combination of the RTE1–RTE3 development sets, and BPI is added for two-way. All models are tested on the RTE4 (TAC 2008) gold set. A named-entity heuristic filter flags pairs that cannot be entailments. A separate SVM then sorts those pairs into contradiction or unknown.

The best results come from MLP trained on RTE3 alone: 58.4% for two-way and 55.4% for three-way. Both beat the baseline. Merging more RTE corpora did not help. This questions the common belief that more training data always improves shallow entailment classifiers.

Key points

• The feature vector is deliberately small to limit overfitting. Its four features are stemmed-string Levenshtein, a token-level Levenshtein distance, WordNet similarity and longest common substring.
• The lexical and WordNet features align tokens of T and H with the Hungarian algorithm.
  • Word senses are chosen with Lesk.
  • Word-to-word similarity is computed from the depth of the least common subsumer.
• The NER filter rule: if H contains a named entity of some type that is absent from T, the pair is labelled non-entailment.
  • It fired on about 10% of RTE4 pairs.
  • It was correct in 66 of 92 cases (0.71).
• On RTE4 test, the best scores were MLP trained on RTE3: 58.4% two-way and 55.4% three-way.
• Ten-fold cross-validation gave much higher scores than the test set, which points to overfitting or dataset shift:
  • 67.85% two-way (RTE3, BayesNet);
  • 65.5% three-way (RTE3, MLP).
• Kappa was low: on average 0.138 two-way and 0.168 three-way, with a best of 0.35. The Levenshtein and lexical-distance features correlate at about 0.86, and dropping either one cost only around 1%.

Interesting ideas

• I1. Hold the feature set fixed and vary only the classifier and the training-corpus combination; this cleanly separates the effect of the learner from the effect of the data.
• I2. A cheap, high-precision veto rule says a hypothesis mentioning an entity of some type missing from the text cannot be entailed; it handles a slice of cases before any statistical model runs.
• I3. Alignment through optimal bipartite matching (Hungarian algorithm) pairs each H token with its best partner in T, which beats a bag-of-words overlap because every token is matched at most once.
• I4. The error analysis shows the filter failing on "fifty years ago" versus "fiftieth birthday"; surface entity matching misses equivalent temporal and numeric expressions.
• I5. Comparing cross-validation scores with scores on a separate test set exposed a large gap between them, a reminder that in-distribution scores flatter shallow models.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Our evaluation runs could freeze inputs and swap one component at a time, such as the parser or the data mix, so that gains can be attributed honestly.
• I2 (answer): Readers answering yes/no questions could first check whether the question names entities absent from the world of the text, and then explain a "not stated" answer by pointing to that missing entity.
• I3 (world): Optimal matching could link entities or frame arguments in a question to those in the text world, giving an explicit, inspectable alignment.
• I4 (world): Normalizing dates and quantities into values in the world layer would avoid such false contradictions and keep the reasoning explainable.
• I5 (learn): Evaluations should always include a held-out corpus from a different source alongside cross-validation, to catch overfitting to one treebank or test collection.