📄 A mostly unlexicalized model for recognizing textual entailment

Summary

The paper addresses the textual entailment half of the FEVER fact-checking task. The system must decide whether retrieved Wikipedia evidence SUPPORTS or REFUTES a claim. The authors worry that models relying on specific words or word embeddings memorize domain-bound cues, which then fail to transfer, for example from news to science. They therefore build a classifier whose inputs are almost entirely unlexicalized:

• overlap ratios for all words, nouns and verbs;
• verb polarity mismatches detected through dependency parses;
• WordNet antonym counts;
• agreement between numbers;
• similarity statistics computed from GloVe embeddings;
• indicator features for short closed lists of hedging and refuting words, whose meaning is assumed stable across domains.

Retrieval reuses the official DrQA-based baseline. Text is processed with CoreNLP, and classification is a linear SVM. The system scores below the lexicalized decomposable attention baseline (FEVER score 0.19 vs 0.2745). However, its transparency supports ablation, weight inspection and error analysis, and these point to missing syntactic and discourse signals. The work is a modest, interpretable step toward fact verification that can move between domains.

Key points

• Data: FEVER training set of 145,449 claims and development set of 19,998. After dropping NOT ENOUGH INFO claims, 109,810 claims were used for training and 13,332 for tuning.
• Test results with baseline retrieval: label accuracy 0.3694 vs 0.4884 for the baseline, and FEVER score 0.1900 vs 0.2745. Evidence F1 is identical (0.1826) because both use the same IR component.
• Oracle setting on the dev set (gold evidence sentences): 55.60 at submission and 56.88 after adding the lexical similarity features, vs 65.13 for the baseline.
• Feature groups:
  • 5 overlap features;
  • 60 hedging indicators;
  • 19 refuting-word indicators;
  • 4 polarity tuples;
  • 4 antonym features;
  • 2 numeric features;
  • embedding similarity statistics (max, min and mean pairwise cosine, plus averaged-vector similarity).
• Ablation: removing word overlap drops accuracy to 50.89, and removing hedging drops it to 50.96. These were the largest effects. The strongest REFUTES weight (1.30) belongs to the case where a verb appearing in both texts carries negation in the claim yet appears affirmatively in the evidence.

Interesting ideas

• I1. Encode polarity disagreement as cross-tuples (claim negated/evidence positive and the reverse) over verbs shared by both texts, ignoring which verb it is; in this paper it became the most decisive refutation signal.
• I2. Measure overlap three ways (relative to the claim, to the evidence, and to their union), because coverage in one direction means something different from coverage in the other.
• I3. Ban content words as features but allow small closed lists of function-like cues (hedges, refuters) whose meaning holds across domains; this keeps the model portable while still capturing stance.
• I4. Treat numbers mentioned in the claim but missing from the evidence as a sign of exaggeration or conflict.
• I5. Use a transparent model to run group-wise ablation, print the top-weighted features per class, and manually trace a sample of errors. The traced errors revealed that missing role alignment (who wrote the film), implicit negation via "only", and argumentative cues such as "despite" misfire on neutral encyclopedic text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): When a claim frame and an evidence frame share a predicate, compare their polarity slots and emit an explicit mismatch flag that can serve as a refutation reason.
• I2 (answer): A reader can check how much of a question's content is covered by the story, separately from how much of the story the question touches, to tell "unsupported" apart from "off-topic".
• I3 (frames): Hedges and refuting markers can be tagged as modality or speech-act attributes on frames, kept separate from the domain lexicon.
• I4 (world): Compare quantities stored as measures and values in the text world, and report a claimed number that has no matching state as a concrete, explainable discrepancy.
• I5 (learn): Mova's evaluation can ablate feature or rule groups and include register-shifted negative controls, such as neutral Wikipedia-style text, to catch cues that only work in argumentative prose.