Elamur Research⌕
☰
home / papers / 2023 / ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

Summary

The paper asks whether a general-purpose LLM can handle interpretive annotation that depends on world knowledge and on inferring what an author intends. Traditionally, such work is left to human coders. The test task is guessing whether a tweet was written by a Democrat or a Republican, where the true party is known. The author sampled 500 tweets from US senators posted in the two months before the 2020 election, drawn from the Twitter parliamentarian database. ChatGPT-4 was given a single zero-shot instruction through the API. Its labels were compared with those from expert coders and from MTurk workers who received the same instructions. ChatGPT-4 was more accurate and more reliable than both human groups, and its bias was equal or lower. It also correctly labeled tweets that required contextual reasoning, such as religious messages aimed at a party's base. The author argues this could make large-scale interpretive text analysis feasible in the social sciences. He also warns that prompt sensitivity, inherited biases and opaque emergent abilities mean every task must be validated.

Key points

  • Data: tweets posted by US senators between 3 September and 3 November 2020, taken from the van Vliet et al. Twitter parliamentarian database.
    • Retweets, replies, tweets with URLs and messages under 100 characters were filtered out.
    • The final sample is balanced: 250 Republican and 250 Democratic tweets.
  • Setup: ChatGPT-4 was queried through the API with one zero-shot prompt asking for an educated guess of "Democrat" or "Republican".
    • It ran 5 times at temperature 0.2 and 5 times at 1.0, for 5000 classifications in total.
  • Results: every individual human coder scored below the model.
    • The model's mean accuracy was almost the same at both temperatures.
    • MTurk answers were evaluated as a per-tweet majority vote.
  • MTurk quality: without strict quality controls, MTurk accuracy in pre-testing was barely distinguishable from chance, so best-practice safeguards were added.
  • Caveats: the author lists prompt sensitivity, reproduced social biases, poorly understood emergent abilities, and the model's failure to signal uncertainty. His recommendation is careful task-specific validation rather than out-of-the-box use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.06588 · arXiv 2304.06588

@misc{tornberg2023chatgpt,
  title = {{ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning}},
  author = {Tornberg, Petter},
  year = {2023},
  doi = {10.48550/arxiv.2304.06588},
  eprint = {2304.06588},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.06588}
}

References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M Weber, Sociological writings. (Continuum International Publishing Group) Vol. 60, (1994). find
  2. A Sayer, Realism and social science. Realism Soc. Sci. pp. 1–224 (1999). find
  3. JS Murthy, GM Siddesh, KG Srinivasa, TwitSenti: a real-time Twitter sentiment analysis and visualization framework. J. Inf. & Knowl. Manag. 18, 1950013 (2019) Publisher: World Scientific. find
  4. W Van Atteveldt, MA Van der Velden, M Boukes, The validity of sentiment analysis: Comparing manual annotation, crowd-coding, dictionary approaches, and machine learning algorithms. Commun. Methods Meas. 15, 121–140 (2021) Publisher: Taylor & Francis. find
  5. M Stubbs, Grammar, text, and ideology: computer-assisted methods in the linguistics of representation. Appl. linguistics 15, 201–223 (1994) Publisher: Oxford University Press. find
  6. J Eisenstein, Introduction to natural language processing. (MIT press), (2019). find
  7. T Brown, et al., Language models are few-shot learners. Adv. neural information processing systems 33, 1877–1901 (2020). find
  8. J Wei, et al., Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022). find our card
  9. S Bubeck, et al., Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712 (2023). find our card
  10. T Kojima, SS Gu, M Reid, Y Matsuo, Y Iwasawa, Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 (2022). find our card
  11. PY Wu, JA Tucker, J Nagler, S Messing, Large Language Models Can Be Used to Estimate the Ideologies of Politicians in a Zero-Shot Learning Setting. arXiv preprint arXiv:2303.12057 (2023). — [arXiv:2303.12057](https://arxiv.org/abs/2303.12057) find
  12. F Gilardi, M Alizadeh, M Kubli, Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056 (2023). find our card
  13. LP Argyle, et al., Out of One, Many: Using Language Models to Simulate Human Samples. arXiv preprint arXiv:2209.06899 (2022). find our card
  14. M Chmielewski, SC Kucker, An mturk crisis? shifts in data quality and the impact on study results. Soc. Psychol. Pers. Sci. 11, 464–473 (2020). find
  15. DM Ziegler, et al., Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 (2019). find
  16. J Schulman, et al., ChatGPT: Optimizing language models for dialogue (2022). find
  17. PW Anderson, More is different. Science 177, 393–396 (1972) Publisher: JSTOR. find
  18. Y Bang, et al., A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). find our card
  19. HG Frankfurt, On bullshit. (Princeton University Press), (2005). find
  20. L Lucy, D Bamman, Gender and representation bias in GPT-3 generated stories in Proceedings of the Third Workshop on Narrative Understanding. pp. 48–55 (2021). find our card
  21. JT Ornstein, EN Blasingame, JS Truscott, How to Train Your Stochastic Parrot: Large Language Models for Political Texts. Work. paper. https://joeornstein.github.io/publications/ornstein-blasingame-truscott.pdf (2022). ↗
  22. A Palmer, A Spirling, Large Language Models Can Argue in Convincing and Novel Ways About Politics: Evidence from Experiments and Human Judgement, (Working paper), Technical report (2023). find
  23. L van Vliet, P Törnberg, J Uitermark, The Twitter parliamentarian database: Analyzing Twitter politics across 26 countries. PloS one 15, e0237073 (2020). find
  24. M Reiss, Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark. OSF Prepr. (2023). find
  25. J Grimmer, BM Stewart, Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Polit. analysis 21, 267–297 (2013). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tornberg-2023-chatgpt-4-outperforms-experts-crowd/