Elamur Research
☰
home / papers / 2020 / Data Augmentation using Pre-trained Transformer Models

Data Augmentation using Pre-trained Transformer Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2003.02245

Summary

Augmenting text data helps classifiers trained on very small datasets. Earlier generators often ignored class labels, so a synthetic sentence could invert its meaning (for example, flip its sentiment). Others relied on architecture-specific tricks, such as CBERT's reuse of BERT segment embeddings, and prior comparisons between methods were not run on equal terms. The authors propose a single generic recipe: fine-tune any pre-trained transformer on training examples with the class label prepended, then generate one synthetic example per original. They test three model families: an auto-encoder (BERT), an auto-regressive model (GPT2) and a seq2seq denoiser (BART). Experiments simulate a low-resource regime of 10 examples per class on SST-2, SNIPS and TREC, using a BERT classifier. Baselines are EDA, back-translation and CBERT. Every family improves accuracy, and the pre-trained seq2seq model performs best overall. The paper also analyses diversity and label preservation and gives practical guidelines. This makes it a useful model-agnostic starting point for augmenting small labeled corpora.

Key points

  • Two labeling schemes are compared. In prepend, the label is split into ordinary subwords. In expand, the label becomes a new vocabulary token, which needed about 150 training epochs for BERT. Plain prepending proved competitive and works for any model.
  • Plain GPT2 generation tends to lose the class. Seeding the prompt with the label plus the first k = 3 words of an original example (GPT2 context) is needed to get useful data.
  • BART is fine-tuned to reconstruct text with 40% of words masked, in two variants: BART word (single-word masks) and BART span (continuous chunks). Word-level masking beat subword masking. The masking ratio also controls output diversity.
  • Type-token ratio analysis shows EDA gives the most diverse tri-grams and back-translation the most diverse unigrams. Seq2seq models balance diversity against semantic fidelity. RoBERTa prepend outperformed BERT prepend on TREC (66.12 vs 64.74 mean accuracy).
  • The authors note two limitations: gains are hard to achieve in rich-resource settings, and no single generator wins on every dataset. Code is available at github.com/varinf/TransformersDataAugmentation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

174 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 162 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2003.02245 · arXiv 2003.02245

@misc{kumar2020data,
  title = {{Data Augmentation using Pre-trained Transformer Models}},
  author = {Kumar, Varun and Choudhary, Ashutosh and Cho, Eunah},
  year = {2020},
  doi = {10.48550/arxiv.2003.02245},
  eprint = {2003.02245},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.02245}
}

References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2019. Not enough data? deep learning to the rescue! arXiv preprint arXiv:1911.03118. — [arXiv:1911.03118](https://arxiv.org/abs/1911.03118) find
  2. Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv… find our card
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  4. Sergey Edunov, Myle Ott, M. Auli, and David Grangier. 2018. Understanding back-translation at scale. ArXiv, abs/1808.09381. find
  5. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. find
  6. Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing. 2019. Learning data manipulation for augmentation and weighting. In Advances in Neural Information Processing Systems, pages 15738–15749. find
  7. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational… find
  8. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858. find our card
  9. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  10. Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. arXiv preprint arXiv:1805.06201. find our card
  11. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. find our card
  12. Varun Kumar, Hadrien Glaude, Cyprien de Lichy, and William Campbell. 2019. A closer look at feature space data augmentation for few-shot intent classification. arXiv preprint arXiv:1910.04176. find
  13. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. find our card
  14. Xin Li and Dan Roth. 2002. Learning question classifiers. In Proceedings of the 19th international conference on Computational linguistics-Volume 1, pages 1–7. Association for Computational Linguistics. find
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  16. George A Miller. 1998. WordNet: An electronic lexical database. MIT press. find
  17. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair’s wmt19 news translation task submission. In Proc. of WMT. find our card
  18. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. find our card
  19. Alec Radford, Karthik Narasimhan, Tim Salimans, Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Lu-and Ilya Sutskever. 2018. Improving language ong, and Quoc V. Le. 2019. Unsupervised data aug-understanding by generative pre-training. URL mentation for consistency training. https://s3-us-west-2. amaz… find
  20. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  21. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  22. Melissa Roemmele, Andrew S Gordon, and Reid Swanson. 2017. Evaluating story generation systems using automated linguistic analyses. In SIGKDD 2017 Workshop on Machine Learning for Creativity. find
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015a. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. find
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015b. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. find
  25. Sam Shleifer. 2019. Low resource text classification with ulmfit and backtranslation. find
  26. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… find
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  28. Jason W Wei and Kai Zou. 2019. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196. find our card
  29. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  30. Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional bert contextual augmentation. In International Conference on Computational Science, pages 84–95. Springer. find