Elamur Research
☰
home / papers / 2021 / Want To Reduce Labeling Cost? GPT-3 Can Help

Want To Reduce Labeling Cost? GPT-3 Can Help

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.findings-emnlp.354.pdf

Summary

Manual annotation for NLP is slow and expensive. Most pseudo-labeling methods are tied to one task and need a sizable labeled seed set. This paper asks whether GPT-3 can serve as a cheap annotator whose outputs train smaller deployable models, as an alternative to running the huge model itself. The authors prompt GPT-3 (Davinci) with a few human-labeled examples to label unannotated data. They then fine-tune RoBERTa large on the labeled data for understanding tasks and PEGASUS for generation tasks. Costs are compared using OpenAI API pricing against crowd-sourcing rates from Google Cloud. They also study mixing GPT-3 and human labels under a fixed budget, and active labeling, in which humans re-check the GPT-3 labels that the model was least confident about. Across nine tasks, GPT-3 labels matched human-label performance at 50–96% lower cost. The student models frequently outperformed few-shot GPT-3, and a theoretical error bound supports this result. Mixed labeling beat either source used alone. This matters for budget-limited projects, especially in low-resource settings.

Key points

  • Benchmarks and models:
    • Nine datasets were used: SST-2, CB, RTE, TREC, AGNews, DBPedia, Gigaword, XSum, and SQuAD (question generation).
    • Prompts used 2/4/8 shots for NLU tasks and 1/2/3 shots for NLG tasks.
  • Cost per label: On SST-2, a GPT-3 label costs roughly $0.002, compared with about $0.11 for a crowd-sourced label.
  • Reported savings at equal performance:
    • 96% on SST-2.
    • 93.8% on Gigaword.
    • About 50–75% on the other tasks.
  • Active labeling:
    • GPT-3's confidence (logit) correlates well with label accuracy. Its top-10% most confident labels reached roughly 95%, 90%, and 95% accuracy on TREC, AGNews, and DBPedia.
    • Having humans re-label the least confident items raised TREC accuracy from 77% to 80% under a $2.2 budget.
  • Limitations:
    • With a large or unlimited budget, labeling everything by hand still gives the best results.
    • Fewer shots are often more cost-efficient for NLU, but 1-shot labeling hurts NLG quality.
    • The authors caution against relying on GPT-3 for high-stakes labeling, such as detecting toxic language.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

127 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 118 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-emnlp.354

@inproceedings{wang2021want,
  title = {{Want To Reduce Labeling Cost? GPT-3 Can Help}},
  author = {Wang, Shuohang and Liu, Yang and Xu, Yichong and Zhu, Chenguang and Zeng, Michael},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.354},
  url = {https://doi.org/10.18653/v1/2021.findings-emnlp.354}
}

References

33 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Attenberg, Prem Melville, and Foster Provost. 2010. A unified approach to active dual supervision for labeling features and examples. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 40–55. Springer. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  3. Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. find
  4. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. find
  5. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107–124. find our card
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  7. Pinar Donmez and Jaime G Carbonell. 2008. Proactive learning: cost-sensitive active learning with multiple imperfect oracles. In Proceedings of the 17th ACM conference on Information and knowledge management, pages 619–628. find
  8. Liat Ein Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020. Active learning for bert: An empirical study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 7949–7962. find
  9. Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato. 2019. Revisiting self-training for neural sequence generation. arXiv preprint arXiv:1909.13788. — [arXiv:1909.13788](https://arxiv.org/abs/1909.13788) find
  10. Woohwan Jung and Kyuseok Shim. 2020. Dual supervision framework for relation extraction with distant supervision and human annotation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6411–6423. find
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  12. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786. find our card
  13. Prem Melville and Vikas Sindhwani. 2009. Active dual supervision: Reducing the cost of annotating examples and features. In Proceedings of the NAACL HLT 2009 workshop on active learning for natural language processing, pages 49–57. find
  14. Subhabrata Mukherjee and Ahmed Awadallah. 2020. Uncertainty-aware self-training for few-shot text classification. Advances in Neural Information Processing Systems, 33. find
  15. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  16. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. find
  17. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  18. Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pages 15–18. find our card
  19. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  20. Burr Settles. 2009. Active learning literature survey. find our card
  21. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631… find
  22. Jialin Song, Yuxin Chen, and Yisong Yue. 2019. A general framework for multi-fidelity bayesian optimization with gaussian processes. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3158–3167. PMLR. find
  23. Shang-Yu Su, Chao-Wei Huang, and Yun-Nung Chen. 2019. Dual supervised learning for natural language understanding and generation. arXiv preprint arXiv:1905.06196. find
  24. Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200–207. find
  25. Yaqing Wang, Subhabrata Mukherjee, Haoda Chu, Yuancheng Tu, Ming Wu, Jing Gao, and Ahmed Hassan Awadallah. 2020. Adaptive self-training for few-shot neural sequence labeling. arXiv preprint arXiv:2010.03680. — [arXiv:2010.03680](https://arxiv.org/abs/2010.03680) find
  26. Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. 2021. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations. find
  27. Yichong Xu, Sivaraman Balakrishnan, Arthur Dubrawski, and Aarti Singh. 2020. Regression with comparisons: Escaping the curse of dimensionality with ordinal information. Journal of machine learning research. find
  28. David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189–196. find our card
  29. Wenpeng Yin. 2020. Meta-learning for few-shot natural language processing: A survey. arXiv preprint arXiv:2007.09604. — [arXiv:2007.09604](https://arxiv.org/abs/2007.09604) find
  30. Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyeong Park. 2021. Gpt3mix: Leveraging large-scale language models for text augmentation. arXiv preprint arXiv:2104.08826. find our card
  31. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328–11339. PMLR. find our card
  32. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of Advances in Neural Information Processing Systems. find our card
  33. Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.09690. find our card