Elamur Research⌕
☰
home / papers / 2022 / Data Augmentation for Intent Classification with Off-the-shelf Large Language Models

Data Augmentation for Intent Classification with Off-the-shelf Large Language Models

Paper · 2022 · Gaurav Sahu, Pau Rodriguez, Issam Laradji et al.

Summary

Building intent classifiers for task-oriented chatbots requires labelled utterances, which are expensive to collect. Earlier LM-based augmentation usually fine-tunes a generator on the scarce task data. This paper skips that step and uses off-the-shelf models as is: four GPT-3 engines (Ada, Babbage, Curie, Davinci) and GPT-J. For each intent, a prompt combines the intent name with its few known examples and ends with an open slot for the model to continue. The generated utterances are added to the training data of a BERT-large classifier. The authors test this on CLINC150, HWU64, Banking77 and SNIPS. They use a full few-shot setting with 10 examples per intent and a partial few-shot setting for comparison with the fine-tuning-based Ex2. EDA serves as an edit-based baseline. Augmentation helps clearly when intents are well separated (CLINC150, SNIPS), and larger generators help more. On fine-grained HWU64 and Banking77 the gains shrink, because generations often drift into a neighbouring intent. The result is a cheap, tuning-free recipe with a clear picture of where it breaks down.

Key points

  • Datasets differ in granularity:
    • CLINC150 has 150 intents across 10 domains, plus out-of-scope examples.
    • HWU64 has 64 intents across 18 domains.
    • Banking77 has 77 intents in a single banking domain.
    • SNIPS has 7 smart-speaker intents.
  • In the full few-shot setup, every intent is cut to K = 10 examples. In the partial few-shot setup, only some intents are limited, and these are augmented up to N, the median number of examples per intent in the original data.
  • The classifier is BERT-large with a linear layer on the [CLS] token. The full original validation set is used for hyperparameter tuning, so that comparisons reflect the quality of the generated data.
  • An oracle classifier trained on the full training data measures fidelity: the share of generated utterances it assigns to the intended seed intent. The same oracle also estimates the gains if a human relabelled the generations. LM-generated data shows higher relabelling potential than EDA output.
  • GPT3Mix-style prompts, which list all class names, are found to be poorly suited to tasks with tens or hundreds of intents. Code is released at github.com/ElementAI/data-augmentation-with-llms.

Interesting ideas

  • I1. Prompt around one class at a time. The prompt contains only the target intent's name and its examples, rather than listing every label. This keeps prompts short and focused, so the approach can scale to label sets with hundreds of classes in any classification task.
  • I2. Measure augmentation quality with an oracle. A classifier trained on plentiful data scores how often generations match their intended label. This separates data faithfulness from downstream accuracy, and the same check works for any synthetic-data pipeline where some gold data exists.
  • I3. Estimate what human relabelling would add. Letting the oracle relabel generated samples shows how much value human correction could unlock. This helps decide whether paying annotators to fix synthetic data is worthwhile, and it can rank augmentation methods (e.g., LLM versus EDA) by their potential rather than their raw output.
  • I4. Use the generator itself as a filter. The same LLM, prompted as a classifier, can reject samples that look like they belong to a confusable intent. Rejecting suspect samples is a different task from assigning labels to new ones. It is a cheap cleaning step, though it requires knowing which competing intents to check against.
  • I5. Expect failure on semantically close classes. The finding that generations drift toward neighbouring labels suggests a practical rule. Before relying on synthetic data, check how fine-grained the label set is, and plan for extra filtering or human review where classes overlap.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WaterlooWaterloo · Canada44 works · 12,783 citations🏢ServiceNowSanta Clara · United States5 works · 849 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Language models & pretraining · 1,343 works Dialogue systems · 401 works

Cite

Cite as Gaurav Sahu, Pau Rodriguez, Issam Laradji, Parmida Atighehchian, David Vazquez and Dzmitry Bahdanau. 2022. Data Augmentation for Intent Classification with Off-the-shelf Large Language Models. Proceedings of the 4th Workshop on NLP for Conversational AI.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.nlp4convai-1.5

@inproceedings{sahu2022data,
  title = {{Data Augmentation for Intent Classification with Off-the-shelf Large Language Models}},
  author = {Gaurav Sahu, Pau Rodriguez, Issam Laradji, Parmida Atighehchian, David Vazquez and Dzmitry Bahdanau},
  booktitle = {Proceedings of the 4th Workshop on NLP for Conversational AI},
  year = {2022},
  doi = {10.18653/v1/2022.nlp4convai-1.5},
  url = {https://doi.org/10.18653/v1/2022.nlp4convai-1.5}
}

References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do Not Have Enough Data? Deep Learning to the Rescue! Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7383–7390. find
  2. Kenton Lee, Kelvin Guu, Luheng He, Tim Dozat, and Hyung Won Chung. 2021. Neural Data Augmentation via Example Extrapolation. arXiv:2102.01335 [cs]. ArXiv: 2102.01335. — [arXiv:2102.01335](https://arxiv.org/abs/2102.01335) find
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… our card
  4. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Pawel Swietojanski Xingkun Liu, Arash Eshghi and Verena Rieser. 2019. Benchmarking natural language understanding services for building conversational agents. In Proceedings of the Tenth International… find
  5. Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT 2018. ArXiv: 1802.05365. our card
  6. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. find
  7. Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyoung Park. 2021. GPT3Mix: Leveraging large-scale language models for text augmentation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2225– 2239, Punta Cana, Dominican Republic. Association for Computat… our card
  8. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference. arXiv:2001.07676 [cs]. ArXiv: 2001.07676. our card
  9. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational L… find
  10. Ivan Vulić, Pei-Hao Su, Samuel Coope, Daniela Gerz, Paweł Budzianowski, Iñigo Casanueva, Nikola Mrkšić, and Tsung-Hsien Wen. 2021. ConvFiT: Conversational fine-tuning of pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 115… find
  11. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/ kingoflolz/mesh-transformer-jax. ↗
  12. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want To Reduce Labeling Cost? GPT-3 Can Help. arXiv:2108.13487 [cs]. ArXiv: 2108.13487. our card
  13. Jason Wei and Kai Zou. 2019. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP… our card
  14. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. our card
  15. Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2018. Conditional BERT Contextual Augmentation. arXiv:1812.06705 [cs]. ArXiv: 1812.06705. — [arXiv:1812.06705](https://arxiv.org/abs/1812.06705) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sahu-2022-data-augmentation-intent-classification-off/