Elamur Research⌕
☰
home / papers / 2023 / ZeroShotDataAug: Generating and Augmenting Training Data with ChatGPT

ZeroShotDataAug: Generating and Augmenting Training Data with ChatGPT

Paper · 2023 · Solomon Ubani, Suleyman Olcay Polat and Rodney Nielsen

Summary

The paper asks whether ChatGPT, prompted without any examples, can supply useful synthetic training data for text classification when labelled data is very scarce. The authors write one short, hand-crafted instruction per class, based on the task, the label description and a handful of training items. Each instruction asks the model to produce 20 new sentences for that class. They test this on SST-2 (Stanford Sentiment Treebank), SNIPS and TREC, keeping only 10 labelled examples per class for training and for development. The zero-shot ChatGPT data is compared with EDA, back-translation, several BERT/GPT-2/BART-based augmenters and few-shot ChatGPT paraphrasing. The zero-shot approach gives the highest accuracy on SST-2 and TREC and essentially ties few-shot ChatGPT on SNIPS. It also keeps improving as more synthetic examples are added. To rule out memorisation, the authors measure how similar the generated text is to the original train and test splits. The work matters because it offers a cheap way to bootstrap classifiers even when no annotated data exists.

Key points

  • Accuracy: ChatGPTzero-shot reached 78.1 on SST-2, 91.2 on SNIPS and 75.3 on TREC. The strongest non-ChatGPT baselines scored 58.0 (BARTword), 87.2 and 67.3 (BARTspan), so the gains are about 20, 4 and 8 points. Few-shot ChatGPT scored 69.6, 91.3 and 66.7.
  • Stability: All runs were repeated 15 times. The zero-shot method showed the smallest variance on SNIPS and TREC.
  • Scaling with more synthetic data: On SST-2, it outperformed every other augmenter at each tested ratio of 1 to 32 synthetic examples per original example, with slower diminishing returns.
  • No original training data: Trained only on generated data, the models scored 0.80 on SST-2, 0.78 on SNIPS and 0.62 on TREC. The SST-2 result beats all augmentation baselines that also had real data.
  • Contamination check: The authors used three similarity measures against the train and test sets: MiniLM sentence-embedding cosine, TF-IDF cosine and percent word overlap. Very few generated items were highly similar, which argues against memorisation. A noted limitation is that prompt design is manual and requires expertise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Solomon Ubani, Suleyman Olcay Polat and Rodney Nielsen. 2023. ZeroShotDataAug: Generating and Augmenting Training Data with ChatGPT. arXiv:2304.14334.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.14334 · arXiv 2304.14334

@misc{ubani2023zeroshotdataaug,
  title = {{ZeroShotDataAug: Generating and Augmenting Training Data with ChatGPT}},
  author = {Solomon Ubani, Suleyman Olcay Polat and Rodney Nielsen},
  year = {2023},
  doi = {10.48550/arxiv.2304.14334},
  eprint = {2304.14334},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.14334}
}

References

15 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. “Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces”. In: arXiv…
  2. Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Zihao Wu, Lin Zhao, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, et al. “Chataug: Leveraging chatgpt for text data augmentation”. In: arXiv preprint arXiv:2302.13007 (2023).
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. “Bert: Pre-training of deep bidirectional transformers for language understanding”. In: arXiv preprint arXiv:1810.04805 (2018).
  4. Varun Kumar, Ashutosh Choudhary, and Eunah Cho. “Data augmentation using pre-trained transformer models”. In: arXiv preprint arXiv:2003.02245 (2020).
  5. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension”. In: arXiv preprint arXiv:1910.13461 (2019).
  6. Xin Li and Dan Roth. “Learning question classifiers”. In: COLING 2002: The 19th International Conference on Computational Linguistics. 2002.
  7. OpenAI. GPT-3.5. [Accessed: 07.04.2023]. url: https://platform.ope nai.com/docs/models/gpt-3-5. ↗
  8. OpenAI. Introducing ChatGPT. [Accessed: 07.04.2023]. url: https://op enai.com/blog/chatgpt. find
  9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. “Language models are unsupervised multitask learners”. In: OpenAI blog 1.8 (2019), p. 9. find
  10. SBERT.net. SentenceTransformers Documentation. [Accessed: 07.04.2023]. url: https://www.sbert.net/index.html. ↗
  11. Rico Sennrich, Barry Haddow, and Alexandra Birch. “Improving neural machine translation models with monolingual data”. In: arXiv preprint arXiv:1511.06709 (2015). find
  12. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. “Recursive deep models for semantic compositionality over a sentiment treebank”. In: Proceedings of the 2013 conference on empirical methods in natural language processing. 2013, pp. 163…
  13. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. “Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers”. In: Advances in Neural Information Processing Systems 33 (2020), pp. 5776–5788. find
  14. Jason Wei and Kai Zou. “Eda: Easy data augmentation techniques for boosting performance on text classification tasks”. In: arXiv preprint arXiv:1901.11196 (2019).
  15. Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. “Conditional bert contextual augmentation”. In: Computational Science– ICCS 2019: 19th International Conference, Faro, Portugal, June 12–14, 2019, Proceedings, Part IV 19. Springer. 2019, pp. 84–95. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ubani-2023-zeroshotdataaug-generating-augmenting-training-data/