📄 Improving the Precision of Natural Textual Entailment Problem Datasets

Summary

The authors ask whether the RTE (Recognizing Textual Entailment) datasets are fit for evaluating precise natural-language inference. Precise inference here means the careful judgement an expert would make, as opposed to loose, plausibility-based reasoning. They drew 150 problems at random from the RTE3 development set, all originally labelled as entailment. Each problem went to three experts in logic or linguistics. The experts re-judged it and wrote down any unstated assumption the entailment depended on, or explained why it failed.

Roughly half of all responses questioned the original "yes". After consolidating the judgements, the authors conclude:

• 28% of the problems do not entail at all.
• 42% entail only once hidden premises are made explicit.
• 30% are fine as they stand.

The result is a small but new kind of resource: 150 precise entailment problems built on naturally written text, with the missing hypotheses spelled out. The study also identifies recurring causes of imprecision and suggests ways to build sharper NLI datasets from the start.

Key points

• Data collection:
  • Each of the 150 RTE3 problems was rated by three experts, each of whom handled a batch of 30 problems.
  • This produced 449 judgements, 146 stated missing hypotheses and 47 justifications for "no".
  • Collection took nearly six months.
• Distribution of responses:
  • 49% plain yes
  • 33% yes conditional on extra hypotheses
  • 7% no without a reason
  • 10% no with a reason
  • Taken together, 50% of responses cast doubt on the original label.
• Agreement (Fleiss' Kappa) was κ=0.33 when answers were grouped as doubtful vs certain, but only κ=0.21 when grouped as yes vs no. Full agreement among the three experts (zero or three doubts) occurred for 48% of problems.
• Final consolidated labels: 42 No, 64 Yes with implicit hypotheses added, 44 plain Yes. Appeals to authority explain more than a quarter of the RTE errors.
• SNLI is argued to be weaker still for precise reasoning. Its pairs are captions of an unseen image, so "contradiction" often means "cannot describe the same picture" rather than logical incompatibility.

Interesting ideas

• I1. Experts often named the same missing premise but split on whether accepting it means "yes" or "no", so a doubtful/certain axis is more reliable than a yes/no label.
• I2. Turning hidden bridging premises (e.g. "making a product implies owning its brand") into explicit, listed hypotheses converts a fuzzy inference into a checkable one.
• I3. Replacing proper names with generic names of the same class separates textual inference from the reader's background knowledge about real entities.
• I4. The paper proposes an asymmetric game: one side writes watertight entailment problems and the other tries to refute them with counterexamples, guided by a list of known error types.
• I5. Many wrong "yes" labels came from treating a reported claim as established fact (an appeal to authority).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): Readers could report entailment in three states (entailed, entailed given a named assumption, not entailed) and be scored on that split rather than a single binary verdict.
• I2 (world): Whenever the world of the text needs a knowledge-graph or ontology fact to close an inference, record that fact as a cited premise in the explanation trace.
• I3 (learn): Use entity-anonymised copies of test items as a negative control, so a drop in accuracy reveals answers that leaked from world knowledge rather than from the text.
• I4 (learn): A self-play loop could pit a problem generator against a refuter, with surviving items becoming silver test data and refuted ones yielding repair rules.
• I5 (frames): Frames for reporting verbs should attribute the embedded proposition to its source as a claim, so the world layer does not assert it as true by default.