📄 Contradictions and Justifications: Extensions to the Textual Entailment Task

Summary

The paper reports on the optional extended task of the RTE-3 challenge. That task added two things to the usual binary entailment decision. First, systems had to give a three-way answer: YES (entailed), NO (contradicts), or UNKNOWN (neither). Second, they had to produce a justification aimed at end users rather than developers. NIST built a three-way answer key for the 800 RTE-3 test pairs, with two independent annotators per pair and adjudication of disagreements. Twelve runs from eight groups were scored by accuracy.

Contradictions made up only about a tenth of the data, and systems handled them poorly. They often missed real contradictions and flagged contradictions where the text actually entailed the hypothesis. Annotator agreement was lower for three labels than for two. As a result, swapping one annotator for another could reorder systems, so an 800-pair set is too small for a reliable three-way comparison.

Six judges rated the justifications for understandability and correctness, and their ratings diverged a lot. The author concludes that explanation evaluation needs to be tied to a real user task before it can be measured quantitatively.

Key points

• Three-way key over 800 pairs: 410 YES (51%), 319 UNKNOWN (40%), 72 NO (9%). The YES set was copied from the two-way key, and the rest was adjudicated at NIST.
• Across 12 runs (9600 responses), systems answered NO 942 times but only 101 of those were correct, which is 12% of the 864 possible correct NOs. By comparison, systems got 50% of YES cases and 61% of UNKNOWN cases right.
• Agreement between NIST judges and the two-way key was 0.90 and 0.91. Three-way agreement between the two NIST judges was 0.83.
• There were 136 disagreements, each worth 0.00125 of accuracy, so the choice of annotator could in principle move a score by up to 0.17. The largest observed gap between runs was 0.063, and this was enough to change system rankings.
• Judges' feedback on justifications:
  • They penalized raw WordNet/DIRT scores, system internals, and jargon such as "polarity" and "hyponym".
  • They also disliked vague filler phrases, chatty templates, and verbatim repetition of the input.
  • Mathematical notation pleased only judges with mathematical training.

Interesting ideas

• I1. Separating "contradicts" from "not stated" and inspecting a full response-by-reference contingency table shows where errors go, for example whether false NOs come mostly from entailed pairs. A single accuracy number hides this.
• I2. A quick bound on how much a score could shift under a different annotator is the number of disagreed items times the per-item weight. Comparing that bound to the gaps between systems tells you whether the test set is big enough to rank them.
• I3. Accuracy conditioned on one gold label rewards systems that default to that label. It must therefore be read alongside overall accuracy, and a run that never outputs a class is a telling case.
• I4. Explanations for users should be short and specific. They should skip steps that are trivially obvious to humans, such as an inventor creating something or a niece being a relative, and they should never expose internal scores or linguistic terminology.
• I5. Ranking systems per item for each rater (ties get the mean rank) and then plotting mean ranks across raters shows directly whether raters disagree about which system is better.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): Our fable and school-test readers should answer yes / no / not stated, and their evaluation should report the full confusion table, so that over-eager contradiction is visible.
• I2 (learn): Before trusting a gain measured on silver data agreed by witness parsers, compute this swing from the witnesses' disagreement count and treat smaller gains as noise.
• I3 (learn): Add a "constant-answer" baseline as a negative control so that per-class scores of readers or repair rules cannot be inflated by a default.
• I4 (back): When generating explanations from the meaning graph, filter out trivial lexical-ontology hops and component scores, and verbalize only the non-obvious inference links in plain English.
• I5 (learn): Use this to compare LLM teacher tiers acting as judges of explanations or parses. Crossing rank lines would show that the judges disagree and that their verdicts should not be pooled blindly.