Elamur Research⌕
☰
home / papers / 2025 / Irony in Emojis: A Comparative Study of Human and LLM Interpretation

Irony in Emojis: A Comparative Study of Human and LLM Interpretation

Paper · 2025 · Yawen Zheng, Hanjia Lyu and Jiebo Luo

Summary

The paper asks whether GPT-4o judges the ironic potential of emojis the way people actually use them. To get a human baseline, the authors take Ciron, a Weibo corpus of about 8,700 posts in which five native Chinese annotators rated each post's irony from 1 to 5. Each emoji inherits the average rating of the posts it appears in, which yields scores for 82 distinct emojis.

GPT-4o is shown each emoji as an image. It is asked, from a social-media-user persona, how likely it would be to pick that emoji to signal irony on an 11-point scale. These answers are rescaled to 1–5. Further runs inject age and gender into the persona.

The model systematically overrates irony relative to human usage, although the two rankings correlate moderately. Varying gender barely changes the model's ratings, while older personas get lower scores.

The study matters because it shows a calibration gap for LLMs used in sentiment tools, chatbots and simulated social agents. It also raises the issue of English-centric training applied to Chinese data.

Key points

  • Human scores come from Ciron (Xiang et al. 2020): about 3,000 of its roughly 8,700 Weibo posts contain emojis. SemEval 2018 English irony data was rejected because only 494 of its tweets have emojis.
  • Most Ciron posts are non-ironic: 49.5% are "not ironic" and 38.7% "unlikely ironic", while only 1.5% are "strongly ironic". For example, the kiss emoji scores 1.22 across 27 posts.
  • GPT-4o was queried three times per emoji at temperature 0.5.
    • Its mean rescaled score is 2.21 (std 1.00) versus 1.73 for humans (std 0.58).
    • A Wilcoxon signed-rank test confirms the model's scores are higher (W = 918.5, p < .001).
  • Despite the level shift, Spearman correlation between model and human scores is significant (ρ = 0.28, p < .05).
  • Persona prompts covered five age bands × two genders. Gender made little difference, while ratings dropped as the stated age rose. Only one model was tested, and English-language validation is left for future work.

Interesting ideas

  • I1. An item-level label (here, an emoji's irony) can be derived by averaging the human ratings of every text containing it, turning sentence annotations into lexical priors without new annotation.
  • I2. The evaluation separates two questions: whether the model ranks items like humans (rank correlation) and whether it sits at the same level (paired test). A system can pass one and fail the other.
  • I3. Asking an LLM how it would use a symbol in the abstract reflects its training-data stereotypes, not actual usage in a given corpus. Comparing the two exposes where the teacher overgeneralizes.
  • I4. Changing a persona attribute in the prompt works as a probe: an attribute that should matter (age) moves the output, while one that perhaps should not (gender) leaves it flat.
  • I5. The model's explanation for an emoji (e.g., the smirk as inherently ambiguous) was checked against real posts where the emoji carried no irony. This shows that an intrinsic-meaning story can contradict actual context.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): the irony module in prag could learn per-word or per-emoji irony priors from sentence-level labels, then report them as explicit evidence in its explanation.
  • I2 (learn): when grading LLM teacher tiers or witness parsers against gold data, check ordering agreement and systematic bias separately, so a well-ranked but miscalibrated teacher is not trusted blindly.
  • I3 (learn): before accepting teacher-generated silver labels for pragmatic phenomena, compare the teacher's context-free judgments with corpus frequencies and down-weight categories it inflates.
  • I4 (learn): perturb prompt attributes that ought to be irrelevant as negative controls, so that any shift in teacher output flags unreliable supervision.
  • I5 (frames): irony detection should rest on explicit incongruity between the emoji or literal frame and the world of the text, with the emoji's general tendency as only a weak prior.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of RochesterRochester · United States36 works · 7,398 citations🎓Chinese University of Hong Kong, ShenzhenShenzhen · China47 works · 2,596 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksSocial Psychology · 306 worksHuman-Computer Interaction · 141 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSentiment & emotion · 972 works

Cite

Cite as Yawen Zheng, Hanjia Lyu and Jiebo Luo. 2025. Irony in Emojis: A Comparative Study of Human and LLM Interpretation. arXiv:2501.11241.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.11241 · arXiv 2501.11241

@misc{zheng2025irony,
  title = {{Irony in Emojis: A Comparative Study of Human and LLM Interpretation}},
  author = {Yawen Zheng, Hanjia Lyu and Jiebo Luo},
  year = {2025},
  doi = {10.48550/arxiv.2501.11241},
  eprint = {2501.11241},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.11241}
}

References

22 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ai, W.; Lu, X.; Liu, X.; Wang, N.; Huang, G.; and Mei, Q. 2017. Untangling emoji popularity through semantic embeddings. In Proceedings of the international AAAI conference on web and social media, volume 11, 2–11. find
  2. Chen, Y.; Yang, X.; Howman, H.; and Filik, R. 2024. Individual differences in emoji comprehension: Gender, age, and culture. Plos one, 19(2): e0297379. ↗
  3. Chen, Z.; Lu, X.; Ai, W.; Li, H.; Mei, Q.; and Liu, X. 2018. Through a gender lens: Learning usage patterns of emojis from large-scale android users. In Proceedings of the 2018 world wide web conference, 763–772. find
  4. Cz˛estochowska, J.; Gligorić, K.; Peyrard, M.; Mentha, Y.; Bień, M.; Grütter, A.; Auer, A.; Xanthos, A.; and West, R. 2022. On the Context-Free Ambiguity of Emoji. Proceedings of the International AAAI Conference on Web and Social Media, 16(1): 1388–1392. find
  5. Garcia, C.; T , urcan, A.; Howman, H.; and Filik, R. 2022. Emoji as a tool to aid the comprehension of written sarcasm: Evidence from younger and older adults. Computers in Human Behavior, 126: 106971. ↗
  6. Guntuku, S. C.; Li, M.; Tay, L.; and Ungar, L. H. 2019. Studying cultural differences in emoji usage across the east and the west. In Proceedings of the international AAAI conference on web and social media, volume 13, 226–235. ↗
  7. Herring, S.; and Dainas, A. 2017. “Nice picture comment!” Graphicons in Facebook comment threads. find
  8. Hu, T.; Guo, H.; Sun, H.; Nguyen, T.-v.; and Luo, J. 2017. Spice up your chat: the intentions and sentiment effects of using emojis. In Proceedings of the International AAAI Conference on Web and Social Media, volume 11, 102–111. find
  9. Kotek, H.; Dockum, R.; and Sun, D. 2023. Gender bias and stereotypes in large language models. In Proceedings of the ACM collective intelligence conference, 12–24. find
  10. Li, W.; Chen, Y.; Hu, T.; and Luo, J. 2018. Mining the relationship between emoji usage patterns and personality. In Proceedings of the international AAAI conference on web and social media, volume 12. find
  11. Lyu, H.; Huang, J.; Zhang, D.; Yu, Y.; Mou, X.; Pan, J.; Yang, Z.; Wei, Z.; and Luo, J. 2024a. GPT-4V(ision) as A Social Media Analysis Engine. ACM Trans. Intell. Syst. Technol. Just Accepted. ↗
  12. Lyu, H.; Qi, W.; Wei, Z.; and Luo, J. 2024b. Human vs. LMMs: Exploring the Discrepancy in Emoji Interpretation and Usage in Digital Communication. In Proceedings of the International AAAI Conference on Web and Social Media, volume 18, 2104–2110. find
  13. Park, J. S.; O’Brien, J.; Cai, C. J.; Morris, M. R.; Liang, P.; and Bernstein, M. S. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, 1–22. find
  14. Qiu, Z.; Qiu, K.; Lyu, H.; Xiong, W.; and Luo, J. 2024. Semantics Preserving Emoji Recommendation with Large Language Models. arXiv preprint arXiv:2409.10760. ↗
  15. Van Hee, C.; Lefever, E.; and Hoste, V. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In Apidianaki, M.; Mohammad, S. M.; May, J.; Shutova, E.; Bethard, S.; and Carpuat, M., eds., Proceedings of the 12th International Workshop on Semantic Evaluation, 39–50. New Orleans, Louisiana: Association for Computational Linguistics.
  16. Wan, Y.; Pu, G.; Sun, J.; Garimella, A.; Chang, K.-W.; and Peng, N. 2023. " kelly is a warm person, joseph is a role model": Gender biases in llm-generated reference letters. arXiv preprint arXiv:2310.09219. ↗
  17. Wang, S. 2022. Sarcastic meaning of the slightly smiling face emoji from Chinese Twitter users: When a smiling face does not show friendliness. International Journal of Languages, Literature and Linguistics, 8(2): 65–73. ↗
  18. Wankhade, M.; Rao, A. C. S.; and Kulkarni, C. 2022. A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7): 5731–5780. ↗
  19. Weissman, B.; and Tanner, D. 2018. A strong wink between verbal and emoji-based irony: How the brain processes ironic emojis during language comprehension. PloS one, 13(8): e0201727. ↗
  20. Xiang, R.; Gao, X.; Long, Y.; Li, A.; Chersoni, E.; Lu, Q.; Huang, C.-R.; et al. 2020. Ciron: a new benchmark dataset for Chinese irony detection.
  21. Zhang, S.; Zhang, X.; Chan, J.; and Rosso, P. 2019. Irony detection via sentiment-based transfer learning. Information Processing & Management, 56(5): 1633–1644. ↗
  22. Zhou, Y.; Xu, P.; Wang, X.; Lu, X.; Gao, G.; and Ai, W. 2024. Emojis decoded: Leveraging chatgpt for enhanced understanding in social media communications. arXiv preprint arXiv:2402.01681. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2025-irony-emojis-comparative-study-human/