Elamur Research
☰
home / papers / 2020 / Good-Enough Compositional Data Augmentation

Good-Enough Compositional Data Augmentation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.676.pdf

Summary

Neural sequence models struggle to transfer familiar pieces of text to new contexts, especially when training data is scarce. Structured grammar-based models do this better, but on real-world data they usually lose to "black boxes". The author proposes GECA, a simple data augmentation rule that does not depend on the model. The method searches the training examples for fragments that occur in at least one identical local environment. Such fragments can also be discontinuous, for example "picks… up" and "puts… down". They are treated as interchangeable, so every other occurrence of the first fragment can be replaced with the second to obtain a new synthetic example. GECA was evaluated on sequence-to-sequence models, on semantic parsing, and on n-gram language models for six languages with small corpora. On the SCAN diagnostic tasks the error rate drops noticeably, and in semantic parsing the result is no worse than the specialized augmentation of Jia and Liang (2016). The paper concludes that compositional generalization can be substantially improved through data alone, without changing the model architecture.

Key points

  • On the SCAN diagnostic tasks GECA reduces the error rate by up to 87%, and in semantic parsing by about 16%. On a data split that tests generalization more strictly, the method outperforms the approach of Jia and Liang (2016).
  • Language modeling was evaluated on English, Kinyarwanda, Lao, Na, Pashto, and Tok Pisin, with corpora ranging from 10K to 2M tokens. The baseline was a 5-gram Kneser–Ney model (5-MKN) in KenLM. Perplexity decreased by about 1% and did not increase for any language.
  • For language modeling, contiguous fragments of up to 2 tokens within a 2-token window were used. New usages were generated only for fragments occurring fewer than 20 times. Separate models trained on the original and the augmented data were interpolated with a weight of 0.05, 0.1, or 0.5.
  • For Kinyarwanda, the method added 913 synthetic sentences to the 3358 existing ones. The author manually checked 100 generated English sentences: 79% of them were grammatical, but only 51% were semantically acceptable.
  • The code is published at github.com/jacobandreas/geca. Among future directions, the author names replacing exact string matching with learned representations and taking constituent boundaries into account.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · AI & machine learning

Cited by

130 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 124 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.676 · arXiv 1904.09545

@inproceedings{andreas2020good,
  title = {{Good-Enough Compositional Data Augmentation}},
  author = {Andreas, Jacob},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.676},
  eprint = {1904.09545},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.676}
}

References

50 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language modeling. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations. find our card
  3. Joost Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. 2018. Jump to better conclusions: Scan both left and right. In Workshop on Analyzing and Interpreting Neural Networks for NLP. find
  4. Jean Berko. 1958. The child’s learning of English morphology. Word. find
  5. Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  6. Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and Jenifer C Lai. 1992. Class-based n-gram models of natural language. Computational Linguistics. find
  7. Ciprian Chelba and Frederick Jelinek. 1998. Exploiting syntactic structure for language modeling. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  8. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 263–270, Ann Arbor, Michigan. find our card
  9. Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT press. find
  10. Alexander Clark. 2000. Inducing syntactic categories by context distribution clustering. In Conference on Natural Language Learning. find
  11. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent neural network grammars. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics. find
  12. Nicholas J Enfield. 2008. A grammar of Lao, volume 38. Walter de Gruyter. find
  13. Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find our card
  14. John Rupert Firth. 1957. A Synopsis of Linguistic Theory, 1930–1955. find
  15. Kuzman Ganchev, Jennifer Gillenwater, Ben Taskar, et al. 2010. Posterior regularization for structured latent variable models. Journal of Machine Learning Research. find
  16. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics. find our card
  17. Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural Turing machines. arXiv preprint arXiv:1410.5401. find
  18. Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. 2015. Learning to transduce with unbounded memory. In Advances in Neural Information Processing Systems. find
  19. Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Workshop on Machine Translation. find our card
  20. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  21. Zhiting Hu, Bowen Tan, Ruslan Salakhutdinov, Tom Mitchell, and Eric P Xing. 2019. Learning data manipulation for augmentation and weighting. arXiv preprint arXiv:1910.12795. — [arXiv:1910.12795](https://arxiv.org/abs/1910.12795) find
  22. Zhiting Hu, Zichao Yang, Ruslan R Salakhutdinov, Lianhui Qin, Xiaodan Liang, Haoye Dong, and Eric P Xing. 2018. Deep generative models with learnable knowledge constraints. In Advances in Neural Information Processing Systems. find
  23. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  24. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics. find our card
  25. Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
  26. Kevin Knight and Jonathan Graehl. 2005. An overview of probabilistic tree transducers for natural language processing. In Proceedings of the International Conference on Intelligent Text Processing and Computational Linguistics, pages 1–24, Mexico City, Mexico. find
  27. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. find
  28. Brenden M Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the International Conference on Machine Learning. find
  29. Brenden M Lake, Tal Linzen, and Marco Baroni. 2019. Human few-shot learning of compositional instructions. arXiv preprint arXiv:1901.04587. find
  30. Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In Proceedings of the International Conference on Learning Representations. find
  31. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  32. Richard Montague. 1973. The proper treatment of quantification in ordinary english. In Approaches to natural language. find
  33. Hermann Ney, Ute Essen, and Reinhard Kneser. 1994. On structuring probabilistic dependences in stochastic language modelling. Computer Speech & Language. find
  34. Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. 2017. Zero-shot task generalization with multi-task deep reinforcement learning. In Proceedings of the International Conference on Machine Learning. find
  35. Alexandre Kimenyi. 1980. A relational grammar of Kinyarwanda, volume 91. Univ. of California Press. find
  36. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics. find
  37. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations. find
  38. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog. find
  39. Dan Klein and Christopher D Manning. 2002. A generative constituent-context model for improved grammar induction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. find
  40. Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. 2017. Learning to compose domain-specific transformations for data augmentation. In Advances in Neural Information Processing Systems, pages 3236–3246. find
  41. Jake Russin, Jason Jo, and Randall C O’Reilly. 2019. Compositional generalization in a deep seq2seq model by separating syntax and semantics. arXiv preprint arXiv:1904.09708. find
  42. Cecilia Summers and Michael J Dinneen. 2019. Improved mixed-example data augmentation. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1262–1270. IEEE. find
  43. Ilya Sutskever, Oriol Vinyals, and Quoc Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. find
  44. Habibullah Tegey and Barbara Robson. 1996. A reference grammar of Pashto. find
  45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. find
  46. John WM Verhaar. 1995. Toward a reference grammar of Tok Pisin: An experiment in corpus linguistics, volume 26. University of Hawaii Press. find
  47. Jason Wei and Kai Zhou. 2019. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  48. John Zelle. 1995. Using Inductive Logic Programming to Automate the Construction of Natural Language Parsers. Ph.D. thesis, Department of Computer Sciences, The University of Texas at Austin. find
  49. Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, pages 658–666. find
  50. Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. In Proceedings of the International Conference on Learning Representations. find our card