📄 A corpus of precise natural textual entailment problems

Summary

Precise inference systems, such as logic-based NLI built on formal semantics, need test data where entailment holds strictly and every hidden assumption is spelled out. The existing resources each fall short. FraCaS is fairly precise but its examples are invented and small. RTE uses real text but treats "very plausible" as good enough to count as entailment.

The authors randomly sampled 150 RTE3 problems labelled as entailment and had each one re-judged by three experts in logic or linguistics. The experts were asked to flag unstated premises and to justify any rejection. The authors then merged these judgments into a corrected set of 150 problems. Each problem carries a new yes/no label and, where relevant, the missing hypotheses made explicit.

The audit shows RTE is unreliable for strict reasoning:

• about 28% of the sampled "yes" items do not hold even with reasonable world knowledge;
• another 42% hold only once extra assumptions are added;
• only about 30% are fine as they stand.

The work matters as a first real-text benchmark for precise entailment. It also provides a catalogue of typical labelling fallacies.

Key points

• The source is the RTE3 test suite: 150 randomly chosen "YES" problems. Three experts judged each problem, and each expert handled 30 problems. Collecting the judgments took almost six months.
• The experts produced 449 judgments, 146 reported missing hypotheses and 47 explanations for negative verdicts. Roughly half of all responses (226, ratio 0.50) cast doubt on the original "yes".
• All three experts agreed on 48% of problems, and a split with exactly one doubting expert was the least frequent outcome. The final labels are 42 "No", 64 "Yes" with added hypotheses and 44 plain "Yes".
• Missing hypotheses were tagged by type: Language (9), Lexicon (15, e.g. buying implies selling) and World (13).
• The main sources of wrong labels are:
  • treating a reported claim as fact: 12 of the 42 errors, essentially an appeal to authority;
  • confusing permission or intention with an actual event: 8;
  • assuming past states still hold: 6;
  • flawed lexical or adjectival reasoning, e.g. "catastrophic floods" taken to mean all floods: 5;
  • pragmatic strengthening of premises.

Interesting ideas

• I1. A source's assertion ("X has warned that…") only entails that X said it, not that it is true; this attribution trap accounts for the largest share of RTE errors.
• I2. Being allowed or intending to do something does not mean it happened, as in "was allowed to leave" versus "left".
• I3. An entailment should come with its bridging assumptions: rather than a bare yes or no, an answer might be a conditional yes that holds only on the assumption that producing a product means holding the brand.
• I4. If background knowledge alone yields the conclusion, the premise is merely priming the reader, so the item does not test entailment at all.
• I5. The authors propose an asymmetric game in which one player builds watertight entailment problems and the other attacks them with counterexamples, guided by the known error types.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): Store reported content as claims attributed to a speaker, kept separate from world facts. Answers then follow from the world only when the source is explicitly trusted.
• I2 (frames): Frames under permission, intention or modal verbs should be marked as non-realized, so they never add a completed event to the world of the text.
• I3 (answer): Readers could report the lexical or world premises they relied on, which keeps every answer explainable and lets graders separate text-based from knowledge-based reasoning.
• I4 (learn): Add a negative control that tries to answer each test item with the premise removed. Items still answered correctly are flagged as knowledge leaks rather than reading successes.
• I5 (learn): This fits self-play, with one side generating premise–hypothesis pairs and the other searching the world model for refuting situations, which yields hard, verified training items.