Elamur Research⌕
☰
home / papers / 2023 / Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech

Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech

Paper · 2023 · Fan Huang, Haewoon Kwak and Jisun An

Summary

The authors ask whether ChatGPT can handle a subjective task that depends on social and cultural context: recognizing implicit hate speech and explaining in natural language (NLE) why a post is hateful. They randomly sampled 795 tweets from the LatentHatred dataset, all labeled implicitly hateful. Each tweet went to ChatGPT (Jan 9, 2023 version) in a fresh session, three times, using a prompt that asks for a yes/no verdict plus a one-sentence justification. The three answers were combined into a single score per tweet. Amazon Mechanical Turk workers then re-judged the cases where ChatGPT disagreed with the dataset labels and rated the quality of explanations. ChatGPT agreed with the original labels about 80% of the time. On the disputed cases, lay annotators mostly sided with ChatGPT, and ChatGPT's explanations pushed their judgments even further toward its verdict. This shows the model's promise for annotation, but also a risk: persuasive explanations could spread wrong labels. The supplied extract stops before the RQ2 results on explanation quality, so those findings are not covered here.

Key points

  • Data: 795 tweets (12.5%, the same size as the original test split) were drawn from the 6,358 implicit hate tweets in LatentHatred. The human-written "implied statements" in that dataset serve as the reference NLEs.
  • Aggregation: each answer was scored +1 (Yes), −1 (No) or 0 (other, e.g. "It's unclear"). The mean of the three answers decided whether a tweet was labeled Hateful, Non-Hateful or Uncertain.
  • Classification results: ChatGPT labeled 636 tweets hateful (80%), 146 non-hateful (18.4%) and 13 uncertain (1.6%).
  • Human re-evaluation of disputed cases: average hatefulness scores were:
    • −0.41 when workers saw only the post;
    • −0.52 when they also saw ChatGPT's explanation (significantly lower, t(316) = −13.75, p<0.001);
    • +0.29 when they saw the human-written NLE.
  • Explanation-quality setup: 100 random tweets were paired with either human or ChatGPT NLEs and rated on 7-point Likert scales for Informativeness and Clarity. Three MTurk Masters rated each pair, and research assistants reviewed the ratings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indiana University BloomingtonBloomington · United States40 works · 4,003 citations🎓Indiana UniversityBloomington · United States42 works · 3,981 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksBias, fairness & safety · 587 works

Cite

Cite as Fan Huang, Haewoon Kwak and Jisun An. 2023. Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech. arXiv:2302.07736.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3543873.3587368 · arXiv 2302.07736

@misc{huang2023is,
  title = {{Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech}},
  author = {Fan Huang, Haewoon Kwak and Jisun An},
  year = {2023},
  doi = {10.1145/3543873.3587368},
  eprint = {2302.07736},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3543873.3587368}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jisun An, Haewoon Kwak, Claire Seungeun Lee, Bogang Jun, and Yong-Yeol Ahn. 2021. Predicting Anti-Asian Hateful Users on Twitter during COVID-19. In EMNLP Findings. find
  2. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter. In 13th International Workshop on Semantic E…
  3. Anja Belz and Eric Kow. 2009. System Building Cost vs. Output Quality in Data-to-Text Generation. In Proceedings of European Workshop on Natural Language Generation. find
  4. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural language inference with natural language explanations. In NeurIPS.
  5. Ona de Gibert, Naiara Perez, Aitor García-Pablos, and Montse Cuadros. 2018. Hate Speech Dataset from a White Supremacy Forum. In Proceedings of the 2nd Workshop on Abusive Language Online.
  6. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2020. Evaluating the state-of-the-art of end-to-end natural language generation: The E2E NLG challenge. Computer Speech & Language (2020). find
  7. Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent Hatred: A Benchmark for Understanding Implicit Hate Speech. In EMNLP.
  8. Engadget. 2023. ChatGPT reportedly reached 100 million users in January. https://www.engadget.com/chatgpt-100-million-users-january-130619073.html. ↗
  9. Aidan Gilson, Conrad Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, and David Chartash. 2022. How Well Does ChatGPT Do When Taking the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. medRxiv (2022), 2022–12. find
  10. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. arXiv:2301.07597 (2023).
  11. Gabriel Emile Hine, Jeremiah Onaolapo, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Riginos Samaras, Gianluca Stringhini, and Jeremy Blackburn. 2017. Kek, cucks, and god emperor trump: A measurement study of 4chan’s politically incorrect forum and its effects on the web. In ICWSM. find
  12. Fan Huang, Haewoon Kwak, and Jisun An. 2023. Chain of Explanation: New Prompting Method to Generate Higher Quality Natural Language Explanation for Implicit Hate Speech. In Companion Proceedings of the ACM Web Conference. find
  13. Katharina Jeblick, Balthasar Schachtner, Jakob Dexl, Andreas Mittermeier, Anna Theresa Stüber, Johanna Topalis, Tobias Weber, Philipp Wesp, Bastian Sabel, Jens Ricke, et al. 2022. ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports. arXiv:2212.14882 (202… — [arXiv:2212.14882](https://arxiv.org/abs/2212.14882) find
  14. Haewoon Kwak, Jeremy Blackburn, and Seungyeop Han. 2015. Exploring cyberbullying and other toxic behavior in team competition online games. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. find
  15. OpenAI. 2022. ChatGPT: Optimizing Language Model for Dialogue. OpenAI blog (2022). find
  16. Joni Salminen, Hind Almerekhi, Milica Milenković, Soon-gyo Jung, Jisun An, Haewoon Kwak, and Bernard J Jansen. 2018. Anatomy of online hate: developing a taxonomy and machine learning models for identifying and classifying hate in online news media. In ICWSM. find
  17. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social Bias Frames: Reasoning about Social and Power Implications of Language. In ACL.
  18. Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023. An Analysis of the Automatic Bug Fixing Performance of ChatGPT. arXiv:2301.08653 (2023). — [arXiv:2301.08653](https://arxiv.org/abs/2301.08653) find
  19. Rohit Sridhar and Diyi Yang. 2022. Explaining Toxic Text via Knowledge Enhanced Text Generation. In NAACL. find
  20. Bowen Zhang, Daijun Ding, and Liwen Jing. 2022. How would Stance Detection Techniques Evolve after the Launch of ChatGPT? arXiv:2212.14548 (2022).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2023-chatgpt-better-than-human-annotators/