Elamur Research⌕
☰
home / papers / 2026 / The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation

The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation

Paper · 2026 · Yuhang Zhou, Yimin Xiao, Wei Ai et al.

Summary

Most work on offensive-language detection focuses on words and largely ignores emojis. Emojis are seldom offensive alone, but they can carry harm through shape-based symbolism, sarcasm, or misuse in context. The authors study this using a random sample of U.S.-geolocated tweets from 2019. They first filtered the tweets with a fine-tuned RoBERTa offensiveness classifier, then confirmed offensive ones with GPT-4. GPT-4 also labelled the role of each emoji, and BERTopic grouped the tweets into offense themes.

Building on this analysis, they designed a multi-step LLM pipeline. It flags emojis likely to cause offense and proposes substitutes that keep the tweet's meaning, rather than rewriting the whole text. Human evaluation showed the pipeline lowers offensiveness better than direct prompting while preserving semantics. The gains, however, appear mainly in mildly offensive tweets. When strong verbal abuse dominates, swapping emojis changes little. The work shows that moderation tools need to treat non-textual signals as a source of harm.

Key points

  • Data: A RoBERTa pre-filter (cardiffnlp/twitter-roberta-base-offensive, probability > 0.5) followed by GPT-4 classification produced 9,285 offensive tweets. A manual check found 91 of 100 sampled tweets were correctly labelled offensive.
  • Emoji roles: Emojis were sorted into four roles: offensive in itself, intensifying, mitigating, or unrelated.
    • Intensifying was the most frequent role (about 5.2k emojis).
    • Human agreement with the GPT-4 role labels was 83%.
    • The top emojis in the "offensive in itself" group did not overlap with those of the other roles.
  • Offense themes: BERTopic, with a minimum cluster size of 20 documents, yielded 14 topics. GPT-4 then mapped these topics to offense categories such as sexual offenses, violence, personal attacks and political attacks.
  • Pipeline effects by offense type:
    • Dehumanizing emojis dropped by 12.8% in personal attacks and 22.5% in political attacks.
    • Body-part symbols dropped by 15.0% in sexual offenses.
    • The direct-prompting baseline instead raised perceived sarcasm (+12.5%) and lowered fluency (−15.0%) on political tweets.
  • Limits: Offensiveness scores fell mostly for tweets with an original score below 3. Changes for highly offensive categories were not statistically significant. The study also covers only English, U.S. tweets from 2019.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksCommunication · 97 worksHuman-Computer Interaction · 141 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 517 works

Cite

Cite as Yuhang Zhou, Yimin Xiao, Wei Ai and Ge Gao. 2026. The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation. Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.nlpcss-1.19

@inproceedings{zhou2026hidden,
  title = {{The Hidden Language of Harm: Examining the Role of Emojis in Harmful Online Communication and Content Moderation}},
  author = {Yuhang Zhou, Yimin Xiao, Wei Ai and Ge Gao},
  booktitle = {Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science},
  year = {2026},
  doi = {10.18653/v1/2026.nlpcss-1.19},
  url = {https://doi.org/10.18653/v1/2026.nlpcss-1.19}
}

References

36 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akiko Aizawa. 2003. An information-theoretic perspective of tf–idf measures. Information Processing & Management, 39(1):45–65. find
  2. Francesco Barbieri, Jose Camacho-Collados, Leonardo Neves, and Luis Espinosa-Anke. 2020. Tweeteval: Unified benchmark and comparative evaluation for tweet classification. arXiv preprint arXiv:2010.12421.
  3. Delphine Battistelli, Cyril Bruneau, and Valentina Dragos. 2020. Building a formal model for hate detection in french corpora. Procedia Computer Science, 176:2358–2365. find
  4. Tommaso Caselli, Valerio Basile, Jelena Mitrović, Inga Kartoziya, and Michael Granitzer. 2020. I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language. In Proceedings of the twelfth language resources and evaluation conference, pages 6193–6202.
  5. Zhenpeng Chen, Xuan Lu, Wei Ai, Huoran Li, Qiaozhu Mei, and Xuanzhe Liu. 2018. Through a gender lens. find
  6. Zhenpeng Chen, Sheng Shen, Ziniu Hu, Xuan Lu, Qiaozhu Mei, and Xuanzhe Liu. 2019. Emoji-Powered representation learning for Cross-Lingual sentiment classification. In WWW 2019. find
  7. Henriette Cramer, Paloma de Juan, and Joel Tetreault. 2016. Sender-intended functions of emojis in us messaging. In MobileHCI 2016. find
  8. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. arXiv preprint arXiv:1905.12516.
  9. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media, volume 11, pages 512–515.
  10. Jiawen Deng, Jingyan Zhou, Hao Sun, Chujie Zheng, Fei Mi, Helen Meng, and Minlie Huang. 2022a. Cold: A benchmark for chinese offensive language detection. arXiv preprint arXiv:2201.06025. find
  11. Yong Deng, Chenxiao Dou, Liangyu Chen, Deqiang Miao, Xianghui Sun, Baochang Ma, and Xiangang Li. 2022b. Beike nlp at semeval-2022 task 4: prompt-based paragraph classification for patronizing and condescending language detection. arXiv preprint arXiv:2208.01312. find
  12. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. EMNLP.
  13. Jing Ge. 2019. Emoji sequence use in enacting personal identity. In Companion Proceedings of The 2019 World Wide Web Conference, WWW ’19, page 426–438, New York, NY, USA. Association for Computing Machinery. find
  14. Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794.
  15. Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel. 2023. The curious decline of linguistic diversity: Training language models on synthetic text. arXiv preprint arXiv:2311.09807. find
  16. Martie G Haselton, Daniel Nettle, and Paul W Andrews. 2015. The evolution of cognitive bias. The handbook of evolutionary psychology, pages 724–746. find
  17. Tianran Hu, Han Guo, Hao Sun, Thuy-vy Nguyen, and Jiebo Luo. 2017. Spice up your chat: the intentions and sentiment effects of using emojis. In ICWSM 2017. find
  18. Xuan Lu, Wei Ai, Xuanzhe Liu, Qian Li, Ning Wang, Gang Huang, and Qiaozhu Mei. 2016. Learning from the ubiquitous language: An empirical analysis of emoji usage of smartphone users. In UbiComp 2016. find
  19. Hanjia Lyu, Weihong Qi, Zhongyu Wei, and Jiebo Luo. 2024. Human vs. lmms: Exploring the discrepancy in emoji interpretation and usage in digital communication. In Proceedings of the International AAAI Conference on Web and Social Media, volume 18, pages 2104–2110. find
  20. Hannah Miller, Jacob Thebault-Spieker, Shuo Chang, Isaac Johnson, Loren Terveen, and Brent Hecht. 2016. “blissfully happy” or “ready tofight”: Varying interpretations of emoji. In Proceedings of the international AAAI conference on web and social media, volume 10, pages 259–268. find
  21. Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2023. Emojis as anchors to detect arabic offensive language and hate speech. Natural Language Engineering, 29(6):1436–1457.
  22. Huy Nghiem and Hal Daumé III. 2024. Hatecot: An explanation-enhanced dataset for generalizable offensive speech detection via large language models. arXiv preprint arXiv:2403.11456. find
  23. OpenAI. 2023. Gpt-4 technical report.
  24. Vishakh Padmakumar and He He. 2023. Does writing with language models reduce content diversity? arXiv preprint arXiv:2309.05196. find
  25. Fan Huang, Haewoon Kwak, and Jisun An. 2023. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. arXiv preprint arXiv:2302.07736. find
  26. Fatemah Husain and Ozlem Uzuner. 2021. A survey of offensive language detection for the arabic language. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 20(1):1–44. find
  27. Shagun Jhaver, Alice Qian Zhang, Quan Ze Chen, Nikhila Natarajan, Ruotong Wang, and Amy X. Zhang. 2023. Personalizing content moderation on social media: User perspectives on moderation choices, interface design, and labor. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW2):1–33. find
  28. Hannah Rose Kirk, Bertram Vidgen, Paul Röttger, Tristan Thrush, and Scott A Hale. 2021. Hatemoji: A test suite and adversarially-generated dataset for benchmarking and detecting emoji-based hate. arXiv preprint arXiv:2108.05921. find
  29. Georgios K Pitsilis, Heri Ramampiaro, and Helge Langseth. 2018. Effective hate-speech detection in twitter data using recurrent neural networks. Applied Intelligence, 48(12):4730–4742. find
  30. Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2021. Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation, 55:477–523. find
  31. Laura Vandenbosch, Johanna MF van Oosten, and Jochen Peter. 2015. The relationship between sexual content on mass media and social media: A longitudinal study. Cyberpsychology, Behavior, and Social Networking, 18(12):697–703. find
  32. Benjamin Weissman and Darren Tanner. 2018. A strong wink between verbal and emoji-based irony: How the brain processes ironic emojis during language comprehension. PloS one, 13(8):e0201727. find
  33. Lingyao Li, Lizhou Fan, Shubham Atreja, and Libby Hemphill. 2023. " hot" chatgpt: The promise of chatgpt in detecting and discriminating hateful, offensive, and toxic comments on social media. arXiv preprint arXiv:2304.10619. — [arXiv:2304.10619](https://arxiv.org/abs/2304.10619) find
  34. Michael Wiegand and Josef Ruppenhofer. 2021. Exploiting emojis for abusive language detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 369–380. find
  35. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint.
  36. Yunze Xiao, Yujia Hu, Kenny Tsu Wei Choo, and Roy Ka-wei Lee. 2024. Toxicloakcn: Evaluating robustness of offensive language detection in chinese with cloaking perturbations. arXiv preprint arXiv:2406.12223. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-hidden-language-harm-examining-role/