📄 A survey on Recognizing Textual Entailment as an NLP Evaluation

Summary

This survey asks how NLP systems should be evaluated and compared. It traces Recognizing Textual Entailment (RTE) as one proposed answer: a shared sentence-pair format in which a system judges whether one sentence is likely to follow from another. Poliak first organizes about thirty years of evaluation practice along two axes. The first is intrinsic versus extrinsic evaluation. The second is general-purpose, phenomenon-oriented test suites versus task-specific held-out corpora scored with accuracy, BLEU or ROUGE. He also covers probing of neural encoders with auxiliary classifiers. He then reviews prominent RTE datasets and newer ones that each isolate a single semantic phenomenon, such as negation, monotonicity, veridicality or presupposition. His central argument is that these fine-grained sets have mostly been used to test entailment models alone. He says they should also be applied to systems trained for other tasks, including generated text and non-neural models. He further recommends graded, multi-annotator labels instead of one categorical label per item. The paper is useful as a map of diagnostic resources and a case for phenomenon-level evaluation.

Key points

• Evaluation is surveyed along two dimensions: intrinsic vs extrinsic, and general-purpose vs task-specific.
  • Oepen and Netter (1995) separate test suites from test corpora.
  • Test suites give more control over the data, systematic coverage, non-redundancy, negative examples and consistent annotation.
  • The two are meant to complement each other.
• Table 3 lists phenomena targeted by recent RTE datasets, with examples:
  • prepositions, comparatives and spatial expressions (Kim et al., 2019);
  • negation and numerical reasoning (Naik et al., 2018; Richardson et al., 2020);
  • monotonicity (Yanaka et al., 2019, 2020);
  • presupposition and implicatures (Jeretic et al., 2020);
  • tense and aspect (Kober et al., 2019);
  • temporal reasoning (Vashishtha et al., 2020).
• The GLUE diagnostics sort items into four coarse classes: word-level meaning (lexical semantics), how predicates relate to their arguments, logical reasoning, and commonsense or world knowledge.
• Few studies have used these sets on non-RTE systems. Two examples:
  • Poliak et al. (2018a) probed a BiLSTM encoder from a neural MT system.
  • Kim et al. (2019) probed encoders trained for CCG parsing, image-caption matching and discourse-marker prediction.
• For generation, the results differ by model:
  • Falke et al. (2019) found that an SNLI-trained entailment model was not accurate enough to rerank abstractive summaries.
  • Barrantes et al. (2020) got clearly better results with transformers trained on Adversarial NLI.
• On labels and naming:
  • Chen et al. (2020) argue for scalar labels.
  • Pavlick and Kwiatkowski (2019) argue for scoring against the full distribution of human judgments, e.g. with cross-entropy, instead of a majority label.
  • The author prefers "RTE" over "NLI" and would ideally call the task Recognizing Textual Inference.

Interesting ideas

• I1. Build test items so that each one exercises exactly one phenomenon, and deliberately include negative items; this gives a per-phenomenon diagnosis instead of a single opaque score.
• I2. Apply entailment-style test sets to systems that were never trained for entailment, turning RTE into a common yardstick across tasks and even non-neural models.
• I3. Use an entailment judge to check generated text and rerank candidate outputs, since a good output should be supported by its source.
• I4. Keep the full spread of annotator judgments as graded labels instead of collapsing them into a majority vote, because real disagreement carries signal.
• I5. Entailment benchmarks mix true entailment with conventional and conversational implicature and world knowledge, which argues for naming these inference types separately.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Organize Mova's evaluation sets by phenomenon (negation, quantifiers, tense) with paired negative controls, so a regression can be traced to one capability.
• I2 (world): Build the world of the text from the premise, query whether the hypothesis holds, and report accuracy separately for veridicality, monotonicity and negation.
• I3 (back): Accept a generated sentence or compressed text only if Mova's own world model shows that it is entailed by the source graph, and use this check to rank alternatives.
• I4 (learn): When witness parsers or LLM teacher tiers disagree on silver data, store the vote distribution and train or score against it rather than discarding the minority view.
• I5 (frames): Tag each inference in prag explanations by its source (lexical entailment, presupposition, implicature or world knowledge), so answers show which kind of reasoning licensed them.