📄 What Knowledge is Needed to Solve the RTE5 Textual Entailment Challenge?

Summary

This working note by Peter Clark, written in 2010 and posted to arXiv in 2018, asks what world and lexical knowledge a machine would need to justify that a text T entails a hypothesis H in the RTE5 challenge. It sets aside surface matching between T and H. Instead, the author hand-analyzes about 20 test pairs, mostly ones his own RTE system BLUE got wrong, and spells out the background facts and inference steps each one requires. One highlighted category is pairs where H shares nearly all its words with T but the gold label is NO. Systems that rely on word overlap are very likely to fail on these. The main knowledge source examined is WordNet glosses, treated as if they were complete and correct logical axioms (they are not). DIRT, DART and PowerSet's Factz database are consulted occasionally. The note's value is a concrete map of the knowledge gap. Even inferences people find trivial, such as linking a flood to a river spilling over its banks, need explicit knowledge that current resources do not reliably supply.

Key points

• The note analyzes roughly 20 RTE5 test pairs, chosen mainly from errors made by the BLUE system and from cases that need world knowledge.
• It focuses on pairs with near-total lexical overlap between T and H but a NO label, which likely defeat all overlap-based RTE systems.
• WordNet glosses are the main hypothetical knowledge source, assumed to be fully axiomatized. DIRT, DART and PowerSet's Factz database are also referenced.
• Example #341 needs the knowledge that a river flooding means it overflows its banks. The author presents this as deceptively easy.
• Each pair is shown twice, in full and in a reduced form that keeps only the relevant parts. The author notes that choosing those parts is itself a hard computational problem.

Interesting ideas

• I1. Do error analysis by asking what knowledge would license the inference, rather than why the score was low; this turns failures into a concrete list of missing facts and rules.
• I2. Treat high-overlap pairs with a NO label as a dedicated stress category, since they expose systems that secretly behave like word matchers.
• I3. Measure a resource's ceiling by imagining it is perfectly formalized (here, WordNet glosses as axioms), which separates "the knowledge isn't there" from "we failed to extract it".
• I4. Reduce each long T to the fragment that actually carries the inference, and treat relevance selection as its own explicit step.
• I5. Anchor the knowledge analysis in concrete, existing resources rather than an idealized ontology, so the conclusions map onto real engineering choices.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): Each failed entailment or QA item could be logged with the specific missing knowledge item (verb class, ontology link, consequence rule), producing a prioritized backlog for the compiled knowledge layer.
• I2 (learn): Build negative-control sets of near-identical sentence pairs with different meanings, and check that Mova's answers change for structural reasons rather than lexical ones.
• I3 (world): Before investing in parsing WordNet, FrameNet or Wikidata entries into rules, hand-check whether an ideal reading of the entries would even cover the failing cases.
• I4 (answer): Readers could first pick the frames in the text world that bear on the question and expose that selection in the explanation, so irrelevant clauses cannot silently affect the answer.
• I5 (frames): When a VerbNet role mapping falls short (for example, flood versus overflow), record which concrete resource entry would have bridged the gap. This keeps fixes in the pragmatics layer traceable.