Elamur Research⌕
☰
home / papers / 2023 / Automated Annotation with Generative AI Requires Validation

Automated Annotation with Generative AI Requires Validation

Paper · 2023 · Nicholas Pangakis; Samuel Wolken; and Neil Fasching

Summary

Generative LLMs are increasingly used in place of human coders for labeling text. Earlier evaluations disagree with each other, relied on few metrics, and often used public benchmarks that may have leaked into model training data. The authors argue that LLM labels should always be checked against a small set of expert human annotations, one task at a time. They propose a five-step loop:

  • write a codebook,
  • have experts label a random sample,
  • prompt the LLM with the same codebook,
  • revise the instructions where errors cluster,
  • measure final quality on held-out human labels.

To test this, they used GPT-4 to reproduce 27 annotation tasks from 11 non-public datasets behind recent social science papers, labeling over 200,000 texts in total. Overall results were good, but quality differed sharply between datasets and even between tasks within one dataset. This variation is why blanket claims about LLM annotators can mislead, and why validation should be a standard step in any annotation project.

Key points

  • Across all tasks, median accuracy was 0.850 and median F1 was 0.707. However, 9 of the 27 tasks had precision or recall below 0.5.
  • The authors introduce a consistency score: each text is classified repeatedly at temperature 0.6, and the modal label is taken as the prediction. Higher agreement across runs strongly predicts a correct label, which makes the score useful for flagging edge cases.
  • One round of human-in-the-loop codebook revision gave only small gains in accuracy and F1. These came mostly from better precision, while recall often dropped.
  • The authors suggest hand-labeling roughly 250–1,250 random samples per task, with more needed for very rare categories. Tooling is released as a Python package (github.com/npangakis/gpt_annotate), and the workflow is LLM-agnostic.
  • Four use cases are outlined:
    • auditing existing human labels,
    • prioritizing texts for manual review,
    • generating training data for a supervised classifier,
    • labeling a whole corpus directly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States221 works · 63,234 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

Cite

Cite as Nicholas Pangakis; Samuel Wolken; and Neil Fasching. 2023. Automated Annotation with Generative AI Requires Validation. arXiv:2306.00176.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.00176 · arXiv 2306.00176

@misc{wolken2023automated,
  title = {{Automated Annotation with Generative AI Requires Validation}},
  author = {Nicholas Pangakis; Samuel Wolken; and Neil Fasching},
  year = {2023},
  doi = {10.48550/arxiv.2306.00176},
  eprint = {2306.00176},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.00176}
}

References

22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Balagopalan, A., Madras, D., Yang, D. H., Hadfield-Menell, D., Hadfield, G. K., and Ghassemi, M. (2023). Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data. Science Advances, 9(19):eabq0701. find
  2. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… — Language Models are Few-Shot Learners
  3. Card, D., Chang, S., Becker, C., Mendelsohn, J., Voigt, R., Boustan, L., Abramitzky, R., and Jurafsky, D. (2022). Computational analysis of 140 years of us political speeches reveals more positive but increasingly polarized framing of immigration. Proceedings of the National Academy of Sciences of t… find
  4. Chmielewski, M. and Kucker, S. C. (2020). An mturk crisis? shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4):464–473. find
  5. Crabtree, B. F. and Miller, W. L. (1992). A template approach to text analysis: Developing and using codebooks. In Doing qualitative research. Sage Publications. find
  6. Cusimano, C. and Goodwin, G. P. (2020). People judge others to have more voluntary control over beliefs than they themselves do. Journal of Personality and Social Psychology, 119. find
  7. Ding, B., Qin, C., Liu, L., Bing, L., Joty, S., and Li, B. (2022). Is gpt-3 a good data annotator?
  8. Douglas, B. D., Ewell, P. J., and Braue, M. (2023). Data quality in online human-subjects research: Comparisons between mturk, prolific, cloudresearch, qualtrics, and sona. PLoS One, 18. find
  9. Gilardi, F., Alizadeh, M., and Kubli, M. (2023). Chatgpt outperforms crowd-workers for text-annotation tasks.
  10. Grimmer, J. and Stewart, B. M. (2013). Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3):267–297. find
  11. He, X., Lin, Z., Gong, Y., Jin, A.-L., Zhang, H., Lin, C., Jiao, J., Yiu, S. M., Duan, N., and Chen, W. (2023). Annollm: Making large language models to be better crowdsourced annotators.
  12. Kennedy, B., Atari, M., Davani, A. M., Yeh, L., Omrani, A., Kim, Y., Coombs, K., Havaldar, S., Portillo-Wightman, G., Gonzalez, E., Hoover, J., Azatian, A., Hussain, A., Lara, A., Olmos, G., Omary, A., Park, C., Wijaya, C., Wang, X., Zhang, Y., and Dehghani, M. (2022). Introducing the gab hate corpu… find
  13. Krippendorff, K. (2018). Content Analysis: An Introduction to Its Methodology. Sage, 4 edition. find
  14. MacQueen, K. M., McLellan, E., Kay, K., and Milstein, B. (1998). Codebook development for team-based qualitative analysis. CAM Journal, 10(2):31–36. find
  15. Neuendorf, K. A. (2016). The Content Analysis Guidebook. Sage Publications. find
  16. Reiss, M. (2023). Testing the reliability of chatgpt for text annotation and classification: A cautionary remark. Working paper. find
  17. Saha, P., Narla, Kalyan, K., and Mukherjee, A. (2023). On the rise of fear speech in online social media. Proceedings of the National Academy of Sciences of the United States of America. find
  18. Schub, R. (2022). Informing the leader: Bureaucracies and international crises. American Political Science Review, 116. find
  19. Spirling, A. (2023). Why open-source generative ai models are an ethical way forward for science. Nature, 413. find
  20. Wang, S., Liu, Y., Xu, Y., Zhu, C., and Zeng, M. (2021). Want to reduce labeling cost? GPT-3 can help. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195–4205, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  21. Zhu, Y., Zhang, P., Haq, E.-U., Hui, P., and Tyson, G. (2023). Can chatgpt reproduce human-generated labels? a study of social computing tasks. find
  22. Ziems, C., Held, W., Shaikh, O., Chen, J., , Zhang, Z., and Yang, D. (2023). Can large language models transform computational social science? Working paper.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pangakis-2023-automated-annotation-generative-ai-requires/