📄 Natural Language Inference from Multiple Premises

Summary

Most entailment benchmarks pair a hypothesis with a single premise. This paper asks what changes when several independent descriptions of one scene must be combined, as with multiple witness reports or posts about the same event. The authors introduce the Multiple Premise Entailment (MPE) task and dataset. Each item has four FLICKR 30K captions of one image as premises and a short hypothesis labelled entailment, neutral or contradiction.

Hypotheses are shortened captions generated using the 2014 denotation graph from Young and colleagues. Any simplification reachable from a premise is excluded, and a cap on word overlap keeps out trivial lexical matches.

Several neural baselines are tested: an LSTM, an attention model, and a "sum of experts" (SE) model that predicts per premise and adds the logits. Annotators also labelled each premise–hypothesis pair separately. This shows that the full-set label often differs from the single-premise labels, so simple voting cannot solve the task. Attention and SE turn out to have complementary strengths.

Key points

• MPE has 10,000 items (8,000 train, 1,000 dev, 1,000 test), each with four premises, one simplified hypothesis and a label from five crowd judgments (Crowdflower and Mechanical Turk).
• Hypothesis overlap with the premises is capped at 0.5 after stopword removal. Mean overlap in training is 0.28, versus 0.44 for SNLI (unlemmatized).
• Label quality:
  • 90% of items had a majority label; the rest had a 2-2-1 split.
  • Ambiguous cases were adjudicated by hand, so 82% of final labels equal the raw majority.
  • Mean annotator agreement with the final label is 0.70.
• Annotators were told to accept hypotheses that "very probably" describe the scene, which matches the Dagan et al. definition rather than SNLI's "definitely true" standard.
• On 100 phenomenon-annotated dev items, results differ by model:
  • Attention does best on word and phrase synonymy and hypernymy.
  • SE does best on mutual exclusion, compatibility and world knowledge.
  • In the dev subset where no single-premise label matches the MPE label (21.8% of items), attention reaches 65.6.

Interesting ideas

• I1. Build the hard negatives from a subsumption hierarchy: generalize the hypothesis upward, then discard any generalization also reachable from a premise, which removes shortcut entailments by construction.
• I2. Collect labels both per premise and for the whole premise set, then bucket accuracy by how many single-premise labels agree with the set label; this exposes where real aggregation is needed.
• I3. Treat independent descriptions as noisy witnesses of one scene that may disagree slightly, and judge a hypothesis against the reconstructed scene rather than any single sentence.
• I4. Use a "most likely true" entailment standard instead of strict logical necessity, explicitly allowing everyday world knowledge.
• I5. Per-premise prediction plus summation (SE) handles mutual exclusion and compatibility well, while word-level attention handles lexical relations; the two are complementary.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): When generating silver entailment or QA probes, Mova could drop candidates its own WordNet-based generalization already derives from the source, so tests measure reasoning rather than hypernym lookup.
• I2 (answer): Fable and test QA could be evaluated by splitting questions into those answerable from one sentence and those needing several, showing whether the world-building step actually pays off.
• I3 (world): Mentions from several sentences could be merged into one entity/state model with tolerance for minor conflicts, and claims checked against that merged world.
• I4 (frames): Implicature handling could mark inferences as probable-by-default, keeping them separate from strict entailments while still using them to answer.
• I5 (world): Check each sentence's frames separately for exclusion conflicts, such as resting versus kayaking, while lexical matching is handled by the ontology, and combine both explainably.