Elamur Research⌕
☰
home / papers / 2019 / What A Sunny Day ☔: Toward Emoji-Sensitive Irony Detection

What A Sunny Day ☔: Toward Emoji-Sensitive Irony Detection

Paper · 2019 · Shirley Anugrah Hayati, Aditi Chaudhary, Naoki Otani et al.

Summary

Irony on social media often depends on emojis. A rainy-weather emoji after a cheerful sentence can flip its meaning. But the SemEval 2018 irony corpus has few ironic tweets that carry emojis, so classifiers trained on it largely ignore them. The authors first profile emoji use in that corpus and find that ironic and non-ironic tweets tend to favour different emojis. They then hand-edit tweets by adding, removing or swapping emojis while leaving the words unchanged. After annotation and validation, this yields a small evaluation set called Imoji. It suggests that emojis mostly create irony through a clash between the polarity of the text and the polarity of the emoji. Building on that observation, they generate training data automatically by appending strongly negative emojis to positive tweets, taken from a sentiment corpus. Adding these examples to the training data of the NTUA-SLP biLSTM model clearly raises recall on Imoji, beyond what an equal amount of extra real irony data achieves. The work offers a cheap way to make irony detectors respond to non-verbal cues.

Key points

  • In the SemEval 2018 irony training data (3,817 tweets), only 406 tweets contain emojis. Among ironic tweets, just 175 of 1,901 (about 9.2%) have one, and 45% of those use several emojis.
  • Manual augmentation covered three cases with 50 sampled tweets each and three annotators, producing 171 rewritten tweets:
    • Case 1: removing or replacing the emoji in an ironic tweet.
    • Case 2: adding an emoji to a non-ironic tweet.
    • Case 3: adding an emoji to an ironic tweet without one.
  • Agreement varied sharply by case: Fleiss' κ was 0.49, 0.02 and 0.23. Making a tweet ironic purely by adding an emoji (Case 2) was the hardest to agree on.
  • The Imoji dataset contains the 100 generated tweets that both validators labelled identically, plus 60 of their source tweets.
  • The automatic pipeline works in three steps:
    • It takes 48 strongly positive and 48 strongly negative emojis from the Emoji Sentiment Ranking.
    • It draws seed tweets from the SemEval 2018 Affect in Tweets data.
    • It appends emojis to the end of each seed, which added roughly 767 training instances.
  • The model is NTUA-SLP (two-layer biLSTM with self-attention and tweet word2vec embeddings), averaged over 10 runs. The control adds the same number of SemEval 2015 irony tweets.

Interesting ideas

  • I1. Build contrast pairs that keep the text fixed and vary only one cue (the emoji), so the cue's effect on the label can be measured in isolation.
  • I2. Treat irony as a clash between two independently scored polarities, here the sentence sentiment against the emoji sentiment, rather than as a holistic text label.
  • I3. Inspect generated samples per seed type before scaling, and drop the unreliable branch. The authors kept only positive-sentiment seeds after seeing that negative text plus a positive emoji often did not read as ironic.
  • I4. Control for data quantity by adding the same number of real instances from another corpus, so gains cannot be blamed on dataset size alone.
  • I5. Report inter-annotator agreement separately for each transformation type to show which phenomena are inherently ambiguous.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Minimal-pair test sets that toggle a single token or marker would show whether a component truly uses that signal or ignores it.
  • I2 (frames): The prag irony module could compare an explicit evaluation in the frame with an attached affect marker and flag a mismatch, with both polarities recorded as the explanation.
  • I3 (learn): Silver-data generators could be audited by template on a small sample, keeping only templates whose outputs are consistently correct.
  • I4 (learn): Any augmentation experiment should include a size-matched negative control before crediting the new data.
  • I5 (answer): Evaluation suites for figurative-language questions could be split by agreement level, so Mova is not penalised on items that humans themselves disagree on.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Psychology · 798 works Social Psychology · 129 works

🧭 Topics 1 Sentiment & emotion · 608 works

Cite

Cite as Shirley Anugrah Hayati, Aditi Chaudhary, Naoki Otani and Alan W Black. 2019. What A Sunny Day ☔: Toward Emoji-Sensitive Irony Detection. Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-5527

@inproceedings{hayati2019what,
  title = {{What A Sunny Day ☔: Toward Emoji-Sensitive Irony Detection}},
  author = {Shirley Anugrah Hayati, Aditi Chaudhary, Naoki Otani and Alan W Black},
  booktitle = {Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)},
  year = {2019},
  doi = {10.18653/v1/d19-5527},
  url = {https://doi.org/10.18653/v1/d19-5527}
}

References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Edward F McQuarrie and David Glen Mick. 1996. Figures of rhetoric in advertising language. Journal of Consumer Research, 22(4):424–438. ↗
  2. Salvatore Attardo, Jodi Eisterhold, Jennifer Hay, and Isabella Poggi. 2003. Multimodal markers of irony and sarcasm. Humor, 16(2):243–260. ↗
  3. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 Task 1: Affect in Tweets. In Proceedings of SemEval, pages 1–17. our card
  4. Francesco Barbieri and Horacio Saggion. 2014. Modelling irony in twitter. In Proceedings of the Student Research Workshop at EACL, pages 56–64. our card
  5. Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends R in Information Retrieval, 2(1–2):1–135. ↗
  6. Christos Baziotis, Athanasiou Nikolaos, Pinelopi Papalampidi, Athanasia Kolovou, Georgios Paraskevopoulos, Nikolaos Ellinas, and Alexandros Potamianos. 2018. Ntua-slp at semeval-2018 task 3: Tracking ironic tweets using ensembles of word and character level attentive rnns. In Proceedings of SemEval, pages 613–621. our card
  7. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. ↗ our card
  8. Christos Baziotis, Nikos Pelekis, and Christos Doulk-eridis. 2017. Datastories at semeval-2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis. In Proceedings of SemEval, pages 747–754. our card
  9. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Language Resources and Evaluation, 47(1):239–268. ↗
  10. Dan Sperber and Deirdre Wilson. 1981. Irony and the use-mention distinction. Philosophy, 3:143–184. find
  11. Zhenpeng Chen, Sheng Shen, Ziniu Hu, Xuan Lu, Qiaozhu Mei, and Xuanzhe Liu. 2019. Emoji-powered representation learning for cross-lingual sentiment classification. In The World Wide Web Conference, pages 251–262. ACM. ↗
  12. Cynthia Van Hee. 2017. Can machines sense irony? find
  13. Cynthia Van Hee. 2018. Exploring the fine-grained analysis and automatic detection of irony on twitter. In Proceedings of LREC. ↗
  14. Carmen Curcó. 2007. Irony: Negation, Echo, and Metarepresentation. Irony in Language and Thought, pages 269–296. find
  15. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2016. Exploring the realization of irony in twitter data. In Proceedings of LREC. our card
  16. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Conference on Empirical Methods in Natural Language Processing (EMNLP). our card
  17. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of SemEval, pages 39– 50. our card
  18. Tony Veale and Yanfen Hao. 2010. Detecting Ironic Intent in Creative Comparisons. In Proceedings of ECAI, pages 765–770. ↗
  19. Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden, and Antonio Reyes. 2015. SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter. In Proceedings of SemEval, pages 470–478. our card
  20. Chuhan Wu, Fangzhao Wu, Sixing Wu, Junxin Liu, Zhigang Yuan, and Yongfeng Huang. 2018. Thu_ngn at semeval-2018 task 3: Tweet irony detection with densely connected lstm and multi-task learning. In Proceedings of SemEval, pages 51–56. find
  21. H Paul Grice. 1975. Logic and conversation. speech acts, ed. by peter cole and jerry morgan, 41-58. find
  22. H Paul Grice. 1978. Further notes on logic and conversation. 1978, 1:13–128. ↗
  23. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Exploring the fine-grained analysis and automatic detection of irony on twitter. Language Resources and Evaluation, 52(3):707–731. ↗
  24. Suzana Ilić, Edison Marrese-Taylor, Jorge A Bal-azs, and Yutaka Matsuo. 2018. Deep contextualized word representations for detecting sarcasm and irony. arXiv preprint arXiv:1809.09795. ↗ our card
  25. Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič. 2015. Sentiment of emojis. PLoS ONE, 10(12):e0144296. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hayati-2019-what-sunny-day-emoji-sensitive/