Elamur Research⌕
☰
home / papers / 2023 / Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Models

Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Models

Paper · 2023 · Nishant Vishwamitra, Keyan Guo, Farhan Tajwar Romit et al.

Summary

Hate speech online tends to surge in bursts tied to real-world events. Each burst brings new targets and new slurs. Conventional moderation tools such as Perspective API struggle with these "new waves" for two reasons. Deciding whether a post is hateful takes nuanced reasoning, for example telling hate apart from criticism or irony. And retraining needs large labeled datasets that do not exist early in a wave.

The authors gathered 31,549 tweets spanning three recent waves, which centered on Russia's 2022 invasion of Ukraine, the 2021 storming of the US Capitol, and COVID-19. They studied how hateful hashtags rise, peak and fade over time, and they showed that common methods for updating existing classifiers fall short.

They then built HateGuard. It uses chain-of-thought (CoT) prompting so that LLMs reason step by step about whether content is hateful. It works zero-shot: newly detected targets and derogatory terms are inserted into the prompts automatically, and the model itself is never retrained.

HateGuard clearly outperformed state-of-the-art detectors. This suggests prompt-level updates could let platforms react to emerging hate quickly.

Key points

  • Dataset: 31,549 tweets covering three event-driven waves, annotated through Amazon Mechanical Turk under IRB approval. The dataset and code are released at github.com/CactiLab/HateGuard.
  • Accuracy: HateGuard scored 10.59% to 88% higher than state-of-the-art models.
  • Late-stage detection: In the final quarter of each wave, it flagged 6.52% to 71.93% more hateful content than the baselines.
  • Term discovery: From only 10 early COVID-19 tweets, the term-extraction step found "antimaskers", "maskhole" and "maskoff". By the end of the Mask-hate buildup phase, this had grown to 39 targets and terms, closely matching the authors' manual analysis.
  • Limitations and future work: The system currently handles English text only. Planned extensions include multilingual encoders, multimodal content such as memes, and tree-of-thought and graph-of-thought reasoning. The paper appeared at IEEE S&P 2024.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University at Buffalo, State University of New YorkBuffalo · United States12 works · 318 citations🎓Clemson UniversityClemson · United States2 works · 28 citations🎓The University of Texas at San AntonioSan Antonio · United States4 works · 25 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Nishant Vishwamitra, Keyan Guo, Farhan Tajwar Romit, Isabelle Ondracek, Long Cheng, Ziming Zhao and Hongxin Hu. 2023. Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Models. arXiv:2312.15099.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/sp54263.2024.00181 · arXiv 2312.15099

@misc{vishwamitra2023moderating,
  title = {{Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Models}},
  author = {Nishant Vishwamitra, Keyan Guo, Farhan Tajwar Romit, Isabelle Ondracek, Long Cheng, Ziming Zhao and Hongxin Hu},
  year = {2023},
  doi = {10.1109/sp54263.2024.00181},
  eprint = {2312.15099},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/sp54263.2024.00181}
}

References

54 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Valdez, M. Ten Thij, K. Bathina, L. A. Rutter, J. Bollen et al., “Social media insights into US mental health during the COVID-19 pandemic: Longitudinal analysis of Twitter data,” Journal of medical Internet research, vol. 22, no. 12, p. e21418, 2020. “Warning incitement of racial, religious hatr… find
  2. “A Tsunami Of Hate: The Covid-19 Hate Speech Pandemic,” https://www.humanrightspulse . com/mastercontentblog/a-tsunami-of-hate-the-covid-19-hate-speech-pandemic, accessed: 2023-12-18. ↗
  3. A. Prabhu, D. Guhathakurta, M. Subramanian, M. Reddy, S. Sehgal, T. Karandikar, A. Gulati, U. Arora, R. R. Shah, P. Kumaraguru et al., “Capitol (pat) riots: A comparative study of twitter and parler,” arXiv preprint arXiv:2101.06914, 2021. — [arXiv:2101.06914](https://arxiv.org/abs/2101.06914) find
  4. P. P. Chiang, “Anti-Asian racism, responses, and the impact on Asian Americans’ lives: a social-ecological perspective,” in COVID-19. Routledge, 2020, pp. 215–229. find
  5. H. A. Choi and O. E. Lee, “To mask or to unmask, that is the question: Facemasks and anti-asian violence during covid-19,” Journal of human rights and social work, vol. 6, no. 3, pp. 237–245, 2021. find
  6. S. Wang, M. Schraagen, E. T. K. Sang, and M. Dastani, “Public sentiment on governmental covid-19 measures in dutch social media,” 2020. find
  7. C. Wardle and E. Singerman, “Too little, too late: social media companies’ failure to tackle vaccine misinformation poses a real threat,” bmj, vol. 372, 2021. find
  8. D. A. Broniatowski, M. Dredze, and J. W. Ayers, ““First Do No Harm”: Effective Communication About COVID-19 Vaccines,” pp. 1055–1057, 2021. find
  9. M. Wypych and M. Bilewicz, “Psychological toll of hate speech: The role of acculturation stress in the effects of exposure to ethnic slurs on mental health among Ukrainian immigrants in Poland.” Cultural diversity and ethnic minority psychology, 2022. find
  10. A. Shevtsov, C. Tzagkarakis, D. Antonakaki, P. Pratikakis, and S. Ioannidis, “Twitter Dataset on the Russo-Ukrainian War,” arXiv preprint arXiv:2204.08530, 2022. — [arXiv:2204.08530](https://arxiv.org/abs/2204.08530) find
  11. “Perspective API,” 2020, https://www.perspectiveapi . com. ↗
  12. “Azure Text Moderation,” https://docs.microsoft . com/en-us/azure/ cognitive-services/content-moderator/text-moderation-api, accessed: 2023-12-188. ↗
  13. “IBM Toxic Comment Classifier,” https://www.ml-exchange . org/ models/max-toxic-comment-classifier/, accessed: 2023-12-18. ↗
  14. “Hate Speech,” https://transparency.fb . com/policies/community-standards/hate-speech/?from=https%3A%2F%2Fm . facebook . com% 2Fcommunitystandards%2Fhate speech%2F&refsrc=deprecated, 2021. ↗
  15. “Hateful conduct policy,” https://help.twitter . com/en/rules-and-policies/hateful-conduct-policy, accessed: 2023-12-18. ↗
  16. T. Davidson, “Machine learning and the sociology of automated hate speech detection,” 2022. find
  17. K. Gelber, Speaking back: The free speech versus hate speech debate. John Benjamins Publishing, 2002, vol. 1. find
  18. J. W. Howard, “Free speech and hate speech,” Annual Review of Political Science, vol. 22, pp. 93–109, 2019. find
  19. “Harmful content can evolve quickly. Our new AI system adapts to tackle it.” https://ai.facebook . com/blog/harmful-content-can-evolve-quickly-our-new-ai-system-adapts-to-tackle-it/, accessed: 2023-12-18. ↗
  20. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 824–24 837, 2022.
  21. K. Thomas, D. Akhawe, M. Bailey, D. Boneh, E. Bursztein, S. Consolvo, N. Dell, Z. Durumeric, P. G. Kelley, D. Kumar et al., “Sok: Hate, harassment, and the changing landscape of online abuse,” in 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021, pp. 247–267. find
  22. “Hate Crime Laws,” https://www.justice . gov/crt/hate-crime-laws, accessed: 2023-12-18. ↗
  23. M. Duggan, “Online harassment 2017.” find
  24. “Microsoft, “Civility, Safety & Interaction Online”,” https://news.microsoft . com/wp-content/uploads/prod/sites/421/2020/ 02/Digital-Civility-2020-Global-Report . pdf, accessed: 2023-12-18. ↗
  25. Y. Qu, X. He, S. Pierson, M. Backes, Y. Zhang, and S. Zannettou, “On the Evolution of (Hateful) Memes by Means of Multimodal Contrastive Learning,” arXiv preprint arXiv:2212.06573, 2022. — [arXiv:2212.06573](https://arxiv.org/abs/2212.06573) find
  26. T. Davidson, D. Warmsley, M. Macy, and I. Weber, “Automated hate speech detection and the problem of offensive language,” in Proceedings of the International AAAI Conference on Web and Social Media, vol. 11, no. 1, 2017, pp. 512–515.
  27. “How Facebook uses super-efficient AI models to detect hate speech,” https://ai.facebook . com/blog/how-facebook-uses-super-efficient-ai-models-to-detect-hate-speech/, accessed: 2023-12-18. ↗
  28. “Twitter Says AI Flags Over Half of Tweets Violating Terms of Services,” https://www.emergingtechbrew . com/stories/2020/07/20/ twitter-says-ai-flags-half-tweets-violating-terms-services, accessed: 2023-12-18. ↗
  29. “Facebook Community Standards Enforcement Report,” https://transparency.fb . com/data/community-standards-enforcement/, accessed: 2023-12-18. ↗
  30. “A healthier twitter: Progress and more to do,” //blog . twitter . com/en us/topics/company/2019/health-update, accessed: 2023-12-18. https: find
  31. “Content moderators at YouTube, Facebook and Twitter see the worst of the web and suffer silently,” https://www.washingtonpost . com/ technology/2019/07/25/social-media-companies-are-outsourcing-their-dirty-work-philippines-generation-workers-is-paying-price/, accessed: 2023-12-18. ↗
  32. K. Dinakar, R. Reichart, and H. Lieberman, “Modeling the detection of textual cyberbullying,” in Proceedings of the International AAAI Conference on Web and Social Media, vol. 5, no. 3, 2011, pp. 11–17. find
  33. W. Warner and J. Hirschberg, “Detecting hate speech on the world wide web,” in Proceedings of the second workshop on language in social media, 2012, pp. 19–26.
  34. J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang, “Learning under concept drift: A review,” IEEE transactions on knowledge and data engineering, vol. 31, no. 12, pp. 2346–2363, 2018. find
  35. G. Widmer and M. Kubat, “Learning in the presence of concept drift and hidden contexts,” Machine learning, vol. 23, pp. 69–101, 1996. find
  36. A. Tsymbal, “The problem of concept drift: definitions and related work,” Computer Science Department, Trinity College Dublin, vol. 106, no. 2, p. 58, 2004. [58] “Cdc museum covid-19 timeline,” https://www.cdc . gov/museum/ timeline/covid19 . html, accessed: 2023-12-18. ↗
  37. M. Omar and D. Mohaisen, “Making Adversarially-Trained Language Models Forget with Model Retraining: A Case Study on Hate Speech Detection,” 2022, pp. 887–893. [59] B. Lichtenstein, “From “coffin dodger” to “boomer remover”: Outbreaks of ageism in three countries with divergent approaches to coronav… find
  38. S. MacAvaney, H.-R. Yao, E. Yang, K. Russell, N. Goharian, and O. Frieder, “Hate speech detection: Challenges and solutions,” PloS one, vol. 14, no. 8, p. e0221152, 2019. [60] “Mask Rules Expand Across U.S. as Clashes Over the Mandates Intensify,” https://www.nytimes . com/2020/07/16/us/coronaviru… ↗
  39. Billy Perrigo, “Facebook Says It’s Removing More Hate Speech Than Ever Before. But There’s a Catch,” 2019. [Online]. Available: https://time.com/5739688/facebook-hate-speech-languages/ [61] “A Timeline of COVID-19 Vaccine Development,” https://www . biospace . com/article/a-timeline-of-covid-19-va… ↗
  40. M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan et al., “Show your work: Scratchpads for intermediate computation with language models,” arXiv preprint arXiv:2112.00114, 2021. [62] David French, “It’s Clear That America Is Deeply…
  41. F. Huang, H. Kwak, and J. An, “Chain of explanation: New prompting method to generate higher quality natural language explanation for implicit hate speech,” arXiv preprint arXiv:2209.04889, 2022. — [arXiv:2209.04889](https://arxiv.org/abs/2209.04889) find
  42. N. Vishwamitra, R. R. Hu, F. Luo, L. Cheng, M. Costello, and Y. Yang, “On analyzing covid-19-related hate speech using bert attention,” in 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, 2020, pp. 669–676. find
  43. W. Wang, V. W. Zheng, H. Yu, and C. Miao, “A survey of zero-shot learning: Settings, methods, and applications,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1–37, 2019. find
  44. L. Yan, Y. Zheng, and J. Cao, “Few-shot learning for short text classification,” Multimedia Tools and Applications, vol. 77, no. 22, pp. 29 799–29 810, 2018. find
  45. L. Stappen, F. Brunn, and B. Schuller, “Cross-lingual zero-and few-shot hate speech detection utilising frozen transformer language models and AXEL,” arXiv preprint arXiv:2004.13850, 2020. — [arXiv:2004.13850](https://arxiv.org/abs/2004.13850) find
  46. M. Mozafari, R. Farahbakhsh, and N. Crespi, “Cross-lingual few-shot hate speech and offensive language detection using meta learning,” IEEE Access, vol. 10, pp. 14 880–14 896, 2022. find
  47. B. AlKhamissi, F. Ladhak, S. Iyer, V. Stoyanov, Z. Kozareva, X. Li, P. Fung, L. Mathias, A. Celikyilmaz, and M. Diab, “ToKen: Task Decomposition and Knowledge Infusion for Few-Shot Hate Speech Detection,” arXiv preprint arXiv:2205.12495, 2022. find
  48. Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung et al., “A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,” arXiv preprint arXiv:2302.04023, 2023.
  49. “2020 Time Capsule \#5: The ‘Chinese Virus’,” https://www.theatlantic . com/notes/2020/03/2020-time-capsule-5-the-chinese-virus/608260/, 2020. ↗
  50. “Urban Dictionary has a new word for coronavirus screw-ups: COVIDIOT,” https://nypost.com/2020/03/24/urban-dictionary-has-anew-word-for-coronavirus-screw-ups-covidiot/, 2020. ↗
  51. “Coronavirus outbreak: What is “COVIDIOTS” trending on Twitter?” https://www.financialexpress . com/lifestyle/coronavirusoutbreak-what-is-covidiots-trending-on-twitter/1907432/, 2020. ↗
  52. Y. Bao, C. Quan, L. Wang, and F. Ren, “The role of pre-processing in twitter sentiment analysis,” in Intelligent Computing Methodologies: 10th International Conference, ICIC 2014, Taiyuan, China, August 3-6, 2014. Proceedings 10. Springer, 2014, pp. 615–624. find
  53. S. Taylor and G. J. Asmundson, “Negative attitudes about facemasks during the covid-19 pandemic: The dual importance of perceived ineffectiveness and psychological reactance,” PLoS One, vol. 16, no. 2, p. e0246317, 2021. find
  54. R. Killick, P. Fearnhead, and I. A. Eckley, “Optimal detection of changepoints with a linear computational cost,” Journal of the American Statistical Association, vol. 107, no. 500, pp. 1590–1598, 2012. [63] E. Winter, “The Russian Siege of the Azovstal Steel Plant in Ukraine: An International Human… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vishwamitra-2023-moderating-new-waves-online-hate/