📄 Annotation Artifacts in Natural Language Inference Data

Summary

The paper examines whether large corpora for natural language inference (NLI) actually require a model to perform logical inference. The SNLI and MultiNLI corpora were created by crowdworkers. Each worker received a premise sentence and wrote three hypotheses for it: one that follows from it (entailment), one neutral (neutral), and one that contradicts it (contradiction). The authors show that this collection method leaves systematic traces in the hypotheses, called annotation artifacts. These make it possible to guess the label without seeing the premise. A fastText classifier trained only on hypotheses correctly identifies the class in about two thirds of SNLI examples and in slightly more than half of MultiNLI examples. An analysis of words via PMI and of sentence length links these traces to annotators' strategies. For entailment, annotators generalize and remove gender or exact quantities; for neutral, they add a purpose or a modifier; for contradiction, they use negation. The test sets were then split into Easy and Hard: Easy contains the examples this classifier recognized, and Hard contains the rest. Strong NLI models perform considerably worse on Hard. Thus, progress in NLI has been overestimated, and the task remains open. This is an important lesson for anyone who collects their own datasets through crowdsourcing.

Key points

• fastText (bag of words and bigrams, premise not taken into account): 67.0% on SNLI, 53.9% on MultiNLI matched, and 52.3% on mismatched. The majority-class baseline yields only about 34–35%. Logistic regression and DAM without the premise gave similar results.
• Words with the highest PMI: for entailment, outdoors, animal, instrument, some, various; for neutral, tall, sad, first, favorite, because; for contradiction, nobody, no, never, nothing, sleeping. Some of these patterns were probably prompted by the example in the SNLI instructions, where dogs turn into "animals outdoors".
• Length in SNLI: the median neutral hypothesis is 9 tokens long, and 60% of entailment hypotheses have at most 7 tokens. When sentences are treated as bags of words, 8.8% of entailment hypotheses are fully contained in the premise. Among neutral and contradiction hypotheses, only 0.2% are.
• The DAM, ESIM, and DIIN models on SNLI: full test set, 84.7, 85.8, and 86.5; Hard, 69.4, 71.3, and 72.7; Easy, 92.4, 92.6, and 93.4. On MultiNLI matched, the Hard portion yields 55.8–64.1, and Easy yields 85.3–87.6.
• The authors recommend not removing artifacts but distributing them more evenly across labels. They propose using the Hard subsets as an additional test for evaluating NLI models.