📄 A Proposal on Evaluation Measures for RTE

Summary

Bergmair examines how systems in the Recognising Textual Entailment (RTE) challenges are scored and argues that the standard measures can mislead. The gold labels are balanced for the two-way task but skewed for the three-way task (ENTAILED / UNKNOWN / CONTRADICTION). This lets trivial systems that copy the label distribution look competitive. The paper also argues that average precision, borrowed from IR, fits RTE poorly. It evaluates a ranking without requiring the system to commit to labels, so a system's ranking and its labels can contradict each other. It also ignores the symmetry between entailment and contradiction that logical negation creates. The author recasts evaluation in information-theoretic terms and proposes mutual information between gold and system labels. Every constant or random baseline scores zero under this measure. Unlike accuracy and Cohen's kappa, it penalises merging an informative label into a vague one. The practical recommendations are:

• stop reporting average precision;
• report mutual information alongside accuracy and, optionally, confidence-weighted score.

Key points

• Within the non-entailed pairs of RTE-4, 70% were UNKNOWN and 30% CONTRADICTION. The RTE-3 PILOT was even more skewed, at 82%/18%.
• A system that labels every pair ENTAILED gets three-way accuracy .500. That beats two thirds of RTE-3 PILOT participants and one third of RTE-4 participants. On RTE-4, random labelling scores .394, all-UNKNOWN .350 and all-CONTRADICTION .150.
• A system output is called "sound" if its labels equal some cutoff of its own ranking. Otherwise it is "self-contradictory": no gold standard could give it 100% on both accuracy and average precision.
• In the worked example, conflating two labels lowers mutual information from 0.0836 to 0.0262 bits. Over the same change, accuracy rises from 44% to 51% and kappa from 0.1277 to 0.1433.
• The example has H(G) = 1.4277, which bounds H(G|L) from above, and the system reaches H(G|L) = 1.3441. If systems choose confidence weights, H(G) should be reported together with I(L;G).

Interesting ideas

• I1. Constant-label and random-label systems serve as reference points, and in skewed three-way data they beat many real systems on accuracy, so a score only means something next to these baselines.
• I2. Mutual information between gold and predicted labels gives exactly zero to any system whose output tells you nothing about the gold label, regardless of class imbalance.
• I3. An informative label should not be merged into a vague one, and accuracy can reward such merging while MI correctly penalises it, so "I don't know" is worth keeping distinct.
• I4. Negation swaps ENTAILED and CONTRADICTION but leaves UNKNOWN unchanged, so a consistent system must respect that symmetry.
• I5. A system's confidence ranking and its committed labels must be consistent, meaning some cutoff of the ranking reproduces the labels, or the output contradicts itself.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Every evaluation run could automatically score constant and random-label negative controls on the same test set and flag any metric where they look respectable.
• I2 (learn): Reporting MI next to accuracy for skewed label sets, such as entailment verdicts or speech-act types, would show whether a classifier is informative or just copying label frequencies.
• I3 (answer): Readers answering school tests or fable QA should keep an explicit "unknown" outcome and be scored so that collapsing it into a guess is not rewarded.
• I4 (world): Negating a hypothesis can automatically generate paired test items, checking that the world model flips entailment and contradiction and keeps unknown pairs unknown.
• I5 (answer): Answer modules that emit both a confidence order and final verdicts could be checked for this soundness property as a cheap internal sanity test.