Elamur Research⌕
☰
home / papers / 2020 / Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data

Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data

Paper · 2020 · William Huang, Haokun Liu and Samuel R. Bowman

Summary

Crowdsourced NLI benchmarks contain shortcuts such as hypothesis-only cues, negation words and lexical overlap. Models can exploit these cues and still fail outside the benchmark's domain. One proposed remedy is counterfactual augmentation: crowdworkers make small edits to seed examples so that the label flips. The authors test whether this kind of training data actually improves generalization for English NLI. They fine-tune roberta-base on three training sets:

  • the 8.3k-example CNLI set from Kaushik et al.,
  • size-matched random samples of plain SNLI,
  • only the 1.7k SNLI seed pairs behind CNLI.

They evaluate on MNLI (out of domain) and on the Naik et al. and GLUE diagnostic sets. CNLI beats the seed-only set, but it does no better than an equally large unaugmented SNLI sample. On the Negation stress test it is worse than both alternatives. An n-gram count suggests the gain over the seeds mostly reflects added lexical variety, and the minimal edits supply less of it than fresh examples. The authors conclude that the current crowdsourcing protocols for counterfactual editing are not worth the cost for training data. They still see value in such data for building evaluation sets.

Key points

  • Training sets: CNLI (8.3k pairs), five independent random 8.3k subsamples of SNLI, and the 1.7k CNLI seed examples.
  • Training setup:
    • roberta-base trained with jiant v2 alpha, 20 runs per condition, up to 20 epochs with early stopping.
    • Random search over learning rate {1e-5, 2e-5, 3e-5}, batch size {32, 64} and seed.
  • Evaluation:
    • Combined matched and mismatched MNLI validation accuracy.
    • Naik et al. stress tests in six categories: spelling errors, mismatched length, negation, overlapping words, numerical reasoning and antonyms.
    • GLUE diagnostic categories, scored with the R3 correlation metric.
    • The McCoy et al. adversarial examples were dropped because all models scored under 3%.
  • Main results: median MNLI accuracy for CNLI roughly matches the SNLI subsamples and exceeds the seed-only models. On the Negation stress set, CNLI is below both baselines.
  • Unique 4-gram types: SNLI 60.2k, CNLI 42.5k, seeds 21.5k. CNLI therefore holds only about 70% of the 2-, 3- and 4-gram variety of an equal-size unaugmented sample.

Interesting ideas

  • I1. When judging any data-enrichment method, compare it against an equal amount of ordinary data drawn several times, not only against the smaller starting set; otherwise the gain may simply come from having more data.
  • I2. Separate two questions: whether extra examples help when stacked on top of the seeds, and whether they beat collecting new examples under a fixed budget.
  • I3. Counting unique n-gram types per training set is a cheap diagnostic that can explain why one dataset helps more than another.
  • I4. Probe for shortcut cues by adding material that keeps the meaning unchanged but contains a trigger word, such as appending a tautology with a negation (e.g. "and false is not true"). A model that changes its answer is relying on the cue.
  • I5. Minimal label-flipping edits may serve better as evaluation contrast sets than as training data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when new silver sentences or self-induced repair rules are added, benchmark them against the same volume of freshly agreed silver data to see if the special procedure earns its cost.
  • I2 (learn): when deciding whether to spend LLM-teacher calls on editing existing sentences or labelling new ones, measure both settings before committing to a pipeline.
  • I3 (tree): count distinct n-grams of words and of dependency relation paths in candidate treebank additions, to check that a new batch actually widens structural coverage rather than repeating near-duplicates.
  • I4 (frames): append vacuous negated clauses to test sentences and verify that polarity in the extracted frames of the main event does not flip.
  • I5 (answer): build small contrast sets for fable and school-test QA, where one word changes the correct answer, to check that each explainable reasoning step tracks the edit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States136 works · 86,390 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as William Huang, Haokun Liu and Samuel R. Bowman. 2020. Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data. Proceedings of the First Workshop on Insights from Negative Results in NLP.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.insights-1.13

@inproceedings{huang2020counterfactually,
  title = {{Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data}},
  author = {William Huang, Haokun Liu and Samuel R. Bowman},
  booktitle = {Proceedings of the First Workshop on Insights from Negative Results in NLP},
  year = {2020},
  doi = {10.18653/v1/2020.insights-1.13},
  url = {https://doi.org/10.18653/v1/2020.insights-1.13}
}

References

9 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 632–642. The Association for Computational Linguistics. ↗
  2. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020. On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines. CoRR, abs/2006.04884. ↗
  3. Aakanksha Naik, Abhilasha Ravichander, Norman M. Sadeh, Carolyn Penstein Rosé, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 2340–2353. Association for Computational Linguistics. ↗
  4. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4885–4901. Association for Computational Linguistics. find
  5. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, *SEM@NAACL-HLT 2018, New Orleans, Louisiana, USA, June 5-6, 2018, pages 180–191. Association for Computational Linguistics. find
  6. Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA). our card
  7. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. ↗ our card
  8. Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Phil Yeres, Jason Phang, Haokun Liu, Phu Mon Htut, , Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Edouard Grave, Najoung Kim, Thibault Févry, Berlin Chen, Nikita Nangia, Anhad Mohananey, Katharina Kann, Shikha Bordia, Nicolas Patry, David Benton, Ellie Pavlick, and Samuel R. Bowman. 2019b. jiant 1.3: A software toolkit for research on general-purpose text understanding models. http://jiant.info/ . ↗
  9. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2020-counterfactually-augmented-snli-training-data/