📄 An Open-Source Package for Recognizing Textual Entailment

Summary

The paper presents EDITS (Edit Distance Textual Entailment Suite), an open-source Java toolkit for Recognizing Textual Entailment (RTE). RTE asks whether a Hypothesis (H) follows from a Text (T). The authors argue that RTE research lacks freely available systems, which makes experiments and prototyping slow and discourages newcomers. EDITS treats entailment as a matter of distance: the closer T and H are, the more likely entailment holds. Each engine computes a normalized score between 0 and 1. The score comes from an edit-distance or similarity algorithm paired with a user-defined XML cost scheme, plus optional entailment and contradiction rules. During training, the system learns a threshold that separates positive from negative pairs. Costs can be tuned with a genetic-algorithm optimizer, and several engines can be merged either linearly or through a Weka classifier. Earlier versions scored about average on RTE-5, ranked first at EVALITA 2009 for Italian, and served as the core of a question-answering system. The release matters because it offers a shared, reproducible baseline environment for the field.

Key points

• Algorithms: Token Edit Distance (Levenshtein over tokens) and Tree Edit Distance (Zhang and Shasha, 1990). There are also five similarity measures turned into distances: Word Overlap, Jaro-Winkler, Cosine Similarity, Longest Common Subsequence and Jaccard Coefficient.
• Cost schemes:
  • Costs are XML files that use a lisp-like condition language.
  • The Simple scheme charges 10 for insertion, 10 for deletion, 0 for an identical substitution and 20 for a different one.
  • The IDF scheme uses IDF values for the 100K most frequent English words, computed on the Brown Corpus.
• Input and models:
  • Input uses the ETAF XML format, at either the token level or the tree level.
  • Plug-ins exist for TreeTagger, Stanford Parser and OpenNLP.
  • RTE 1-3 and EVALITA 2009 corpora come pre-annotated with the package.
  • A trained model is a zip file holding the threshold, configuration, cost scheme and rules.
• Results:
  • On the RTE-5 2-way task, EDITS reached 60.17% accuracy, against a participant average of 60.36%. A newer model reaches 61.83%.
  • On the RTE-5 Search task it reached 33.44% F1, ranking 3rd of 8.
  • At EVALITA 2009 it ranked first with 71.0% accuracy.
• In the QALL-ME project, EDITS was used for entailment-based relation extraction over 14 CINEMA relations. It reached 72.9% F1, which let the system answer 83% of 400 test questions correctly.

Interesting ideas

• I1. At test time EDITS outputs a decision and a confidence score, plus the exact sequence of edit operations behind the score, so every verdict can be inspected.
• I2. Edit costs can be asymmetric: when texts are much longer than hypotheses, cheap deletion stops short hypotheses that are fully contained in T from being penalized.
• I3. Tree Edit Distance runs over dependency nodes, with costs defined by conditions on the two nodes being compared, so the distance reflects syntactic structure rather than a bag of words.
• I4. Relation extraction is recast as entailment: the input is tested against a small set of textual realizations for each target relation.
• I5. A trained model is packaged together with everything that produced it: threshold, configuration, cost scheme and rules. This makes experiments easy to share and reproduce.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): When a reader checks whether a candidate answer follows from a fable, it could log the alignment steps (kept, dropped and substituted nodes) as the human-readable justification.
• I2 (answer): Matching a short question frame against a long story could make unmatched story material nearly free, while making content missing from the question expensive.
• I3 (learn): A cost-weighted tree edit distance between the outputs of witness parsers could give a graded disagreement score for filtering silver data, rather than a plain match-or-mismatch.
• I4 (frames): Each frame could carry a few canonical template sentences, and a sentence would be linked to the frame whose templates it entails with the lowest structural cost.
• I5 (learn): Bundling each trained parser or repair-rule set with its config and evaluation controls would make regressions traceable and runs replicable.