Elamur Research
☰
home / papers / 2023 / ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2303.15056

Summary

Research projects in NLP and computational social science need labeled texts to train classifiers and to evaluate models. Typically these labels come from trained research assistants, who are costly, or MTurk crowd-workers, who are cheaper but less reliable. The authors tested whether ChatGPT (gpt-3.5-turbo, zero-shot) can replace crowd-workers. They used 6,183 English tweets and news articles about content moderation and US politics. These texts had already been labeled by trained political science students for a previous study, which served as the gold standard. The tasks were relevance, stance, topic and frame detection. The same codebooks were given unchanged to ChatGPT and to filtered MTurk workers. ChatGPT was run twice at each of two temperature settings. Its accuracy was on average about 25 percentage points higher than MTurk's, and its run-to-run consistency was higher than that of both human groups. It also cost roughly thirty times less. These results suggest that LLMs could make large-scale annotation far cheaper and might undermine the business model of crowdsourcing platforms.

Key points

  • Data: four samples were used. These were 2,382 content-moderation tweets (2020–2021), 1,856 US Congress tweets (2017–2022), 1,606 LexisNexis news articles and 339 English tweets from January 2023. The 2023 sample was added to check for memorization.
  • Tasks: relevance (two variants), stance on Section 230 (three classes), topics (six classes), problem/solution frames and 14 policy frames from Card et al. (2015).
  • Intercoder agreement averaged about 56% for MTurk and 79% for trained annotators. For ChatGPT it was 91% at temperature 1 and 97% at temperature 0.2. The lower temperature improved consistency without hurting accuracy.
  • Zero-shot relevance accuracy was 70% for content-moderation tweets, 81% for news articles, 83% for Congress tweets and 59% for the 2023 tweets. The weaker 2023 result came from mislabeling tweets about user suspensions, which shows how much prompt quality matters.
  • ChatGPT cost less than $0.003 per annotation. Replication materials are available on Harvard Dataverse (doi:10.7910/DVN/PQYF6M).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

368 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 354 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1073/pnas.2305016120 · arXiv 2303.15056

@misc{gilardi2023chatgpt,
  title = {{ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks}},
  author = {Gilardi, Fabrizio and Alizadeh, Meysam and Kubli, Maël},
  year = {2023},
  doi = {10.1073/pnas.2305016120},
  eprint = {2303.15056},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1073/pnas.2305016120}
}

References

14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alizadeh, Meysam, Fabrizio Gilardi, Emma Hoes, K Jonathan Klüser, Mael Kubli and Nahema Marchal. 2022. “Content Moderation As a Political Issue: The Twitter Discourse Around Trump’s Ban.” Journal of Quantitative Description: Digital Media 2. find
  2. Argyle, Lisa P., Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting and David Wingate. 2023. “Out of One, Many: Using Language Models to Simulate Human Samples.” Political Analysis pp. 1–15. find our card
  3. Bayerl, Petra Saskia and Karsten Ingmar Paul. 2011. “What Determines Inter-Coder Agreement in Manual Annotations? A Meta-Analytic Investigation.” Computational Linguistics 37(4):699–725. find
  4. Benoit, Kenneth, Drew Conway, Benjamin E. Lauderdale, Michael Laver and Slava Mikhaylov. 2016. “Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data.” American Political Science Review 116(2):278–295. find
  5. Binz, Marcel and Eric Schulz. 2023. “Using cognitive psychology to understand GPT-3.” Proceedings of the National Academy of Sciences 120(6):e2218523120. find
  6. Card, Dallas, Amber Boydstun, Justin H Gross, Philip Resnik and Noah A Smith. 2015. The media frames corpus: Annotations of frames across issues. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language… find
  7. Chmielewski, Michael and Sarah C. Kucker. 2020. “An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results.” Social Psychological and Personality Science 11(4):464–473. find
  8. Desmond, Michael, Evelyn Duesterwald, Kristina Brimijoin, Michelle Brachman and Qian Pan. 2021. Semi-automated data labeling. In NeurIPS 2020 Competition and Demonstration Track. PMLR pp. 156–169. find
  9. Emerson, Guy, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh and Shyam Ratan, eds. 2022. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Seattle: Association for Computational Linguistics. find
  10. Huang, Fan, Haewoon Kwak and Jisun An. 2023. “Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech.” arXiv preprint arXiv:2302.07736 . find
  11. Kojima, Takeshi, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo and Yusuke Iwasawa. 2022. “Large language models are zero-shot reasoners.” arXiv preprint arXiv:2205.11916 . find our card
  12. Kuzman, Taja, Igor Mozetič and Nikola Ljubešić. 2023. “ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification.” arXiv e-prints pp. arXiv–2303. find
  13. Nay, John J. 2023. “Large Language Models as Corporate Lobbyists.”. find
  14. Wu, Patrick Y., Joshua A. Tucker, Jonathan Nagler and Solomon Messing. 2023. “Large Language Models Can Be Used to Estimate the Ideologies of Politicians in a Zero-Shot Learning Setting.”. find