Elamur Research⌕
☰
home / papers / 2024 / Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels

Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels

Paper · 2024 · Nicholas Pangakis and Samuel Wolken

Summary

Supervised text classifiers in computational social science (CSS) usually need human-annotated training data, which is costly and slow to produce. Prompting a large generative model to label an entire corpus is an alternative, but it becomes expensive at scale. The authors test a knowledge-distillation workflow with four steps:

  1. Tune and validate GPT-4 few-shot prompts on 250 human-labeled texts.
  2. Have GPT-4 label 1,000 more texts.
  3. Fine-tune compact "student" classifiers (BERT, RoBERTa, DistilBERT) on those labels.
  4. Evaluate the students against 1,000 held-out human-labeled items.

They replicate 14 English classification tasks from recent CSS articles in high-impact journals. These datasets are kept in password-protected archives, so they are less likely to appear in LLM training data. Students trained on GPT-4 labels perform only slightly below those trained on human labels and nearly match GPT-4 few-shot itself. Researchers can therefore use an LLM to build training data instead of labeling everything with it. Human validation remains essential, because students inherit the LLM's mistakes and biases.

Key points

  • Median F1 across tasks: 0.641 for DistilBERT and 0.624 for BERT trained on 1,000 human labels, 0.592 for GPT-4 few-shot, and 0.586 for BERT trained on 1,000 GPT-4 labels. The median gap between GPT-label and human-label models is 0.039.
  • Students trained on GPT-4 labels differ from GPT-4 few-shot by a median F1 of just 0.006.
  • LLM-derived models favor recall over precision. GPT-4 few-shot reaches a median recall of 0.80, and BERT/DistilBERT on GPT labels reach 0.746. By contrast, BERT trained on human labels has precision 0.214 higher than its GPT-label counterpart.
  • Ablation studies examine:
    • noisy GPT labels;
    • re-running GPT-4 six months later to check for drift;
    • XLNet and Mistral-7B, including Mistral-7B as an open-source few-shot baseline.
  • Cost motivation: classifying 6.2M tweets on four dimensions with GPT-4 alone would cost roughly $9,000. The approach distills a very large teacher (estimated at about 1.7T parameters) into BERT Base, which has about 110M parameters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States221 works · 63,234 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Nicholas Pangakis and Samuel Wolken. 2024. Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels. arXiv:2406.17633.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.17633 · arXiv 2406.17633

@misc{pangakis2024knowledge,
  title = {{Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels}},
  author = {Nicholas Pangakis and Samuel Wolken},
  year = {2024},
  doi = {10.48550/arxiv.2406.17633},
  eprint = {2406.17633},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.17633}
}

References

51 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. 2024. Artifacts or abduction: How do llms answer multiple-choice questions without the question?
  2. Berfu Büyüköz, Ali Hürriyetoğlu, and Arzucan Özgür. 2020. Analyzing elmo and distilbert on socio-political news classification. In Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020, pages 9–18. find
  3. Dallas Card, Serina Chang, Chris Becker, Julia Mendelsohn, Rob Voigt, Leah Boustan, Ran Abramitzky, and Dan Jurafsky. 2022. Computational analysis of 140 years of us political speeches reveals more positive but increasingly polarized framing of immigration. Proceedings of the National Academy of Sci… find
  4. Lingjiao Chen, Matei Zaharia, and James Zou. 2023a. How is chatgpt’s behavior changing over time? find
  5. Zhikai Chen, Haitao Mao, Hongzhi Wen, Haoyu Han, Wei Jin, Haiyang Zhang, and Hui Liuand Jiliang Tang. 2023b. Label-free node classification on graphs with large language models (llms). find
  6. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937. find
  7. Michael Chmielewski and Sarah C. Kucker. 2020. An mturk crisis? shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4):464–473. find
  8. Sayantan Dasgupta, Trevor Cohn, and Timothy Baldwin. 2023. Cost-effective distillation of large language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7346–7354, Toronto, Canada. Association for Computational Linguistics. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Le, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  10. Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq Joty, and Boyang Li. 2022. Is gpt-3 a good data annotator?
  11. Justin Grimmer, Margaret E. Roberts, and Brandon Stewart. 2022. Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton University Press. find
  12. Justin Grimmer and Brandon M. Stewart. 2013. Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3):267–297. find
  13. Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2023. Annollm: Making large language models to be better crowdsourced annotators.
  14. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network.
  15. Daniel J. Hopkins, Yphtach Lelkes, and Samuel Wolken. 2024. The rise of and demand for identity-oriented media coverage. American Journal of Political Science. find
  16. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  17. Ross Deans Kristensen-McLachlan, Miceal Canavan, Márton Kardos, Mia Jacobsen, and Lene Aarøe. 2023. Chatbots are not reliable text annotators. find
  18. David MJ Lazer, Alex Pentland, Duncan J Watts, Sinan Aral, Susan Athey, Noshir Contractor, Deen Freelon, Sandra Gonzalez-Bailon, Gary King, and Helen Margetts. 2020. Computational social science: Obstacles and opportunities. Science, 369(6507):1060–1062. find
  19. Benjamin D. Douglas, Patrick J. Ewell, and Markus Braue. 2023. Data quality in online human-subjects research: Comparisons between mturk, prolific, cloudresearch, qualtrics, and sona. PLoS One, 18. find
  20. Changmao Li and Jeffrey Flanigan. 2023. Task contamination: Language models may not be few-shot anymore. find
  21. Naoki Egami, Christian J. Fong, Justin Grimmer, Margaret E. Roberts, and Brandon M. Stewart. 2022. How to make causal inferences using texts. Science Advances, 8(42):eabg2652. find
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
  23. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text-annotation tasks.
  24. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. find
  25. Jonas Golde, Patrick Haller, Felix Hamborg, Julian Risch, and Alan Akbik. 2023. Fabricator: An open source toolkit for generating labeled training data with teacher llms. find
  26. Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, page 1789–1819. find
  27. Wes McKinney. 2011. pandas: a foundational python library for data analysis and statistics. Python for high performance and scientific computing, 14(9):1–9. find
  28. Jonathan Mellon, Jack Bailey, Ralph Scott, James Breckwoldt, and Marta Miori. 2022. Does gpt-3 know what the most important issue is? using large language models to code open-text social survey responses at scale. Working paper. find
  29. Stefan Müller. 2022. The temporal focus of campaign communication. The Journal of Politics, 84(1):585– 590. find
  30. Kimberly A. Neuendorf. 2016. The Content Analysis Guidebook. Sage Publications. find
  31. OpenAI. 2023. Gpt-4 technical report.
  32. Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. 2023. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. Proceedings of the 40th International… find
  33. Nicholas Pangakis, Samuel Wolken, and Neil Fasching. 2023. Automated annotation with generative ai requires validation.
  34. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, and Trevor Killeen et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems.
  35. Hao Peng, Daniel M. Romero, and Emoke-Agnes Horvat. 2022. Dynamics of cross-platform attention to retracted papers. Proceedings of the National Academy of Sciences, 119(25):585–590. find
  36. Michael Reiss. 2023. Testing the reliability of chatgpt for text annotation and classification: A cautionary remark. Working paper. find
  37. Christopher Michael Rytting, Taylor Sorensen, Lisa Argyle, Ethan Busby, Nancy Fulda, Joshua Gubler, and David Wingate. 2023. Towards coding social science datasets with language models. find
  38. Punyajoy Saha, Narla, Komal Kalyan, and Animesh Mukherjee. 2023. On the rise of fear speech in online social media. Proceedings of the National Academy of Sciences of the United States of America. find
  39. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. find
  40. Maximilian Schreiner. 2023. Gpt-4 architecture, datasets, costs and more leaked. Blog post. find
  41. Saurabh Srivastava, Annarose M B, Anto P V au2, Shashank Menon, Ajay Sukumar, Adwaith Samod T, Alan Philipose, Stevin Prince, and Sooraj Thomas. 2024. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. find
  42. Zhanna Terechshenko, Fridolin Linder, Vishakh Padmakumar, Michael Liu, Jonathan Nagler, Joshua A Tucker, and Richard Bonneau. 2020. A comparison of methods in political science text classification: Transfer learning language models for politics. Working Paper. find
  43. Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. From humans to machines: can chatgpt-like llms effectively replace human annotators in nlp tasks. Workshop Proceedings of the 17th International AAAI Conference on Web and Social Media. find
  44. Petter Törnberg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning.
  45. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks.
  46. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? gpt-3 can help.
  47. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison and´ Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin… — Transformers: State-of-the-Art Natural Language Processing
  48. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2020. Xlnet: Generalized autoregressive pretraining for language understanding.
  49. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. International Conference on Machine Learning, page 11328–11339.
  50. Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. 2023. Can chatgpt reproduce human-generated labels? a study of social computing tasks. find
  51. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large language models transform computational social science? Working paper.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pangakis-2024-knowledge-distillation-automated-annotation-supervised/