Elamur Research⌕
☰
home / papers / 2020 / Syntactic Data Augmentation Increases Robustness to Inference Heuristics

Syntactic Data Augmentation Increases Robustness to Inference Heuristics

Summary

BERT fine-tuned on MNLI scores well on the standard test data but largely ignores word order. On the HANS challenge set, for example, it treats a sentence as entailing a version of itself with subject and object swapped. The authors compare two explanations. Under the Representational Inadequacy Hypothesis, the pretrained representations lack the needed syntax. Under the Missed Connection Hypothesis, the syntax is present in the representations, but MNLI offers too few examples showing that inference depends on it.

To test these, they built small augmentation sets by transforming MNLI sentences with subject/object inversion or passivization, plus a randomly shuffled control. They added each set to the MNLI training data and fine-tuned BERT on the result. The strongest method was inversion applied to the original hypothesis. With about a thousand added examples, it sharply improved HANS accuracy on non-entailed overlap cases without lowering MNLI accuracy. The gains carried over to constructions never seen in augmentation, which largely supports the Missed Connection view. Passives stayed difficult, which suggests real representational gaps in that area.

Key points

  • Augmentation set sizes were 101, 405 and 1215 examples, compared with 297k MNLI training pairs. Roughly 400 inverted examples, about 0.1% of MNLI, already gave large gains.
  • With the large inversion + TRANSFORMED HYPOTHESIS set, HANS lexical-overlap non-entailment accuracy rose from 0.28 to 0.73. MNLI development accuracy stayed at the baseline level of 0.84.
  • The improvement transferred to other structures:
    • relative clauses: 0.33 → 0.83
    • prepositional phrases: 0.41 → 0.89
    • NP/S subsequence cases: 0.02 → 0.50
  • Accuracy on HANS items where the heuristic gives the right answer dropped somewhat. Subsequence and constituent cases improved only modestly, and passive sentences remained weak.
  • Evaluation used 5 seeds per condition, 15 seeds for the best strategy, and a 100-run unaugmented baseline. Data and code are released at github.com/aatlantise/syntactic-augmentation-nli.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

77 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.212

@inproceedings{min2020syntactic,
  title = {{Syntactic Data Augmentation Increases Robustness to Inference Heuristics}},
  author = {Min, Junghyun and McCoy, R. Thomas and Das, Dipanjan and Pitler, Emily and Linzen, Tal},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.212},
  url = {https://doi.org/10.18653/v1/2020.acl-main.212}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations. find
  2. Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don’t take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language… find
  3. Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller, Samuel J. Gershman, and Noah D. Goodman. 2018. Evaluating compositionality in sentence embeddings. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 1596– 1601, Madison, WI. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  5. Yoav Goldberg. 2019. Assessing BERT’s syntactic abilities. arXiv preprint arXiv:1901.05287. find our card
  6. He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 132–142, Hong Kong, China. Association for Computational Linguistics. find our card
  7. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo… find our card
  8. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031. Association for Computational Linguistics. find our card
  9. Juho Kim, Christopher Malon, and Asim Kadav. 2018. Teaching syntax by adversarial distraction. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 79–84, Brussels, Belgium. Association for Computational Linguistics. find
  10. Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pap… find
  11. Rabeeh Karimi Mahabadi and James Henderson. 2019. Simple but effective techniques to reduce biases. arXiv preprint arXiv:1909.06321. — [arXiv:1909.06321](https://arxiv.org/abs/1909.06321) find
  12. R. Thomas McCoy, Junghyun Min, and Tal Linzen. 2019a. BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. arXiv preprint arXiv:1911.02969. find
  13. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019b. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computation… find our card
  14. Pasquale Minervini and Sebastian Riedel. 2018. Adversarially regularising neural NLI models to integrate logical background knowledge. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 65–74, Brussels, Belgium. Association for Computational Linguistics. find
  15. Mehrad Moradshahi, Hamid Palangi, Monica S. Lam, Paul Smolensky, and Jianfeng Gao. 2019. HUBERT Untangles BERT to Improve Transfer across NLP Tasks. arXiv preprint arXiv:1910.12647. — [arXiv:1910.12647](https://arxiv.org/abs/1910.12647) find
  16. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial NLI: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599. find
  17. Deric Pang, Lucy H. Lin, and Noah A. Smith. 2019. Improving natural language inference with a pre-trained parser. arXiv preprint arXiv:1909.08217. — [arXiv:1909.08217](https://arxiv.org/abs/1909.08217) find
  18. Luis Perez and Jason Wang. 2017. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.04621. — [arXiv:1712.04621](https://arxiv.org/abs/1712.04621) find
  19. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  20. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  21. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856–865, Melbourne, Australia. Association for C… find
  22. Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom. 2019. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373. find
  23. Alon Talmor and Jonathan Berant. 2019. MultiQA: An empirical investigation of generalization and transfer in reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4911–4921, Florence, Italy. Association for Computational Linguistics. find our card
  24. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In International Conference on Lear… find our card
  25. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce… find
  26. Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP… find our card
  27. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  28. Yadollah Yaghoobzadeh, Remi Tachet, T. J. Hazen, and Alessandro Sordoni. 2019. Robust natural language inference models with example forgetting. arXiv preprint arXiv:1911.03861. — [arXiv:1911.03861](https://arxiv.org/abs/1911.03861) find
  29. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019. HELP: A dataset for identifying shortcomings of neural models in monotonicity reasoning. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/min-2020-syntactic-data-augmentation-increases-robustness/