📄 BIUTEE: A Modular Open-Source System for Recognizing Textual Entailment

Summary

BIUTEE tackles a practical barrier in Recognizing Textual Entailment (RTE): building a full inference system from scratch takes a lot of work, and the few public systems (EDITS, NutCracker) could use only narrow kinds of knowledge. The authors release an open-source, modular platform. It decides whether a hypothesis H follows from a text T by rewriting T's parse tree into H through a chain of tree edits, called a proof. Edits include entailment rules (lexical, lexical-syntactic, syntactic), replacing a mention with a coreferent one, and "on-the-fly" operations with no backing resource, such as inserting nodes or moving subtrees. Each edit gets a cost: a learned weight vector multiplied by the edit's features. A search procedure looks for the cheapest proof, and the system reports entailment when the total cost falls below a learned threshold. Preprocessors, knowledge resources, learners and new edit types can all be swapped through interfaces or plug-ins. A visual debugger, Tracer, shows every step of the process. BIUTEE scored above the median on RTE-6 and RTE-7 and was the top system on RTE-6.

Key points

• Preprocessing bundles the Easy-First parser, the Stanford NER, the ArkRef coreference resolver, a sentence splitter and number normalization; users can plug in their own tools.
• Proof cost is c(O) = w · Σ f(oᵢ). Both w and the threshold b are trained with Logistic Regression, and the final cost is mapped to a score in [0, 1].
• F1 results:
  • RTE-6: 49.09, against a median of 33.72 and a best competing score of 48.01.
  • RTE-7: 42.93, against a median of 39.89 and a best of 48.00.
• A plug-in implements interfaces that do three things: detect applicable transformations, apply them, and build their feature vectors. A plug-in can also add computations to a global pre-proof phase, such as marking negation and its scope.
• Tracer has two modes. Automatic mode replays the search's proof with per-step costs and before/after trees. Manual mode lists every matching transformation, including ones the search rejected, so the user can apply them one at a time.

Interesting ideas

• I1. Entailment is treated as a derivation: a sequence of tree rewrites whose cost is a sum of learned per-step costs, so every verdict comes with an itemized, inspectable justification.
• I2. Edits that no knowledge resource supports are still allowed but priced by features such as word frequency from a unigram model and named-entity status, so a proof always exists and its weak spots are expensive and visible.
• I3. One rule format (a left-hand and a right-hand subtree) covers lexical substitutions, predicate-argument paraphrases like accept → was offered, and syntactic alternations like active/passive.
• I4. A human can build a proof by hand to locate the fault. If the hand-built proof is cheaper than the system's, the search is at fault. If it is linguistically better but costs more, the learned weights or the training data are at fault.
• I5. The debugger shows the candidate operations that were rejected, not only the winning path.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): yes/no and "does the text say X" questions could be answered by rewriting the text's UD tree toward the question's tree, returning the costed step list as the explanation.
• I2 (world): when linking a text to compiled knowledge, unsupported bridging assumptions could be admitted as explicitly costed steps instead of silently failing or silently succeeding.
• I3 (frames): VerbNet-based role extraction could run after a pass of LHS/RHS subtree rules that normalize alternations, so each frame records which rules produced it.
• I4 (learn): the same contrast could serve as an evaluation probe that tells search errors apart from model or data errors in the parser and in frame building.
• I5 (tree): a trace view for the perceptron parser or gparse that lists competing arcs with their scores at each decision would make parse errors much easier to diagnose.