Elamur Research⌕
☰
home / papers / 2018 / #NonDicevoSulSerio at SemEval-2018 Task 3: Exploiting Emojis and Affective Content for Irony Detection in English Tweets

#NonDicevoSulSerio at SemEval-2018 Task 3: Exploiting Emojis and Affective Content for Irony Detection in English Tweets

Paper · 2018 · Endang Wahyu Pamungkas and Viviana Patti

Summary

The paper reports a SemEval-2018 Task 3 system for spotting irony in English tweets. It covers a binary irony/non-irony decision (Task A) and a four-way typing (Task B): verbal irony arising from contrasting polarity, verbal irony of other kinds, situational irony, or none. The authors use an RBF-kernel SVM over hand-engineered features of three kinds:

  • Twitter-specific structural cues, such as hashtags, mentions, links, quotes, repeated letters and part-of-speech counts.
  • Affective scores drawn from several sentiment and emotion lexicons.
  • New emoji features, including a measure of mismatch between the sentiment of the words and the sentiment of the emojis.

For each submission, ablation picked the feature subset that worked best. Task B was split into a three-stage cascade, and the minority classes were topped up with external #irony tweets. In the constrained Task A, the system placed 10th of 44. In Task B it placed 9th of 32. It was the strongest entry in the unconstrained track of both tasks. Telling situational irony apart from other verbal irony stayed hard. This shows how far shallow affective cues can reach on irony, and where they stop working.

Key points

  • Data: the organizers released 3,834 training tweets and 784 test tweets, originally collected via #irony, #sarcasm and #not. Those hashtags were stripped from the release. In Task B, about half the tweets are non-ironic, and the remainder splits roughly 25% / 13% / 12% across the three irony types.
  • Affective resources:
    • AFINN, in a normalized 0–1 version
    • Emolex, with 14,182 words and Plutchik's eight emotions
    • EmoSenticNet, with Ekman emotions via WordNet-Affect
    • LIWC, using only PosEMO and NegEMO
    • DAL, with 8,742 words scored for Pleasantness, Activation and Imagery
    • the Emoji Sentiment Ranking by Novak et al.
    • VADER, for text polarity
  • Official F1 scores: 0.6216 on Task A against a best of 0.7054, and 0.4131 on Task B against a best of 0.5074. Implementation used scikit-learn, and the code is on GitHub.
  • Effect of external data: adding balanced external corpora (1,500 ironic plus 1,500 non-ironic, and 500 tweets per hashtag) gave no gain on Task A. Extra #irony tweets did help the second Task B stage, which separates polarity-contrast irony from the rest.
  • Error analysis: Twitter symbols were strong signals for non-irony. Emolex helped Task B stage 2 and EmoSenticNet helped stage 3. Situational irony was mostly misclassified (only 8 of 70 correct). Many errors depended on context, such as a link or the removed hashtag.

Interesting ideas

  • I1. Irony is detected as a sentiment clash between two channels of one message, the words versus the emojis, rather than from either channel alone. This gives a concrete and checkable signal of non-literal intent.
  • I2. The fine-grained label set is broken into a cascade: first ironic or not, then polarity-contrast or other, then verbal or situational. Each step can use its own features and its own diagnosis.
  • I3. Extra training data was selected on linguistic grounds: only #irony tweets were added, because earlier analysis found polarity reversal less typical there. Meanwhile, simply adding more data did not help Task A.
  • I4. Ablation was run separately for each submission and each pipeline step, so different subproblems ended up with different feature sets.
  • I5. The error analysis traced many failures to missing context, such as a dead link or a stripped hashtag, rather than to weak features.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): crate prag could compare the polarity of the literal proposition with the polarity of attached signals (emoji, intensifiers, overstatement words) and record any clash as explicit evidence for an irony implicature.
  • I2 (frames): irony handling could follow the same staged order (detect, then type), so every stage leaves an inspectable decision instead of one opaque four-way label.
  • I3 (learn): silver data could be filtered by the property a stage actually needs, and each data addition checked against a no-addition baseline as a negative control.
  • I4 (learn): evaluation could ablate evidence sources per decision type, showing which lexicons or cues truly carry each pragmatic judgment.
  • I5 (answer): readers could flag questions whose correct reading depends on context absent from the text, keeping those apart from genuine model errors in evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TurinTurin · Italy27 works · 6,505 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works Social Psychology · 129 works

🧭 Topics 2 English · 1,373 works Sentiment & emotion · 608 works

Cite

Cite as Endang Wahyu Pamungkas and Viviana Patti. 2018. #NonDicevoSulSerio at SemEval-2018 Task 3: Exploiting Emojis and Affective Content for Irony Detection in English Tweets. Proceedings of the 12th International Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1106

@inproceedings{pamungkas2018nondicevosulserio,
  title = {{\#NonDicevoSulSerio at SemEval-2018 Task 3: Exploiting Emojis and Affective Content for Irony Detection in English Tweets}},
  author = {Endang Wahyu Pamungkas and Viviana Patti},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1106},
  url = {https://doi.org/10.18653/v1/s18-1106}
}

References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Salvatore Attardo. 2006. Irony. Encyclopedia of Language & Linguistics, 6:26–28. (tw2) ”#Consensus of Absolute Hilarious -#MichaelMann to lecture on #Professional #Ethics for #Climate #Scientists? http://t.co/pD0TEMq1Z0” ↗
  2. Salvatore Attardo. 2007. Irony as relevant inappropriateness. In H. Colston and R. Gibbs, editors, Irony in language and thought: A cognitive science reader, pages 135–172. Lawrence Erlbaum. The first tweet is featured by situational irony and was originally including a #not hashtag before the link. Also for humans it is very difficult to get the 5 “Non dicevo sul serio” (Italian) means: ’I didn’t mean that”. find
  3. Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič. 2015. Sentiment of emojis. PloS one, 10(12):e0144296. ↗
  4. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014. Modelling sarcasm in twitter, a novel approach. In Proc. of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 50–58. our card
  5. James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates, 71(2001):2001. find
  6. Delia Irazú Hernańdez Farı́as, Viviana Patti, and Paolo Rosso. 2016. Irony detection in twitter: The role of affective content. ACM Transactions on Internet Technology (TOIT), 16(3):19. find
  7. Robert Plutchik. 2001. The nature of emotions human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American scientist, 89(4):344– 350. find
  8. Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden, and Antonio Reyes. 2015. Semeval-2015 task 11: Sentiment analysis of figurative language in twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 470–478. our card
  9. Soujanya Poria, Alexander Gelbukh, Amir Hussain, Newton Howard, Dipankar Das, and Sivaji Bandyopadhyay. 2013. Enhanced senticnet with affective labels for concept-based opinion mining. IEEE Intelligent Systems, 28(2):31–38. ↗
  10. Raymond W. Gibbs and Herbert L. Colston, editors. 2007. Irony in language and thought. Routledge (Taylor and Francis), New York. find
  11. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on Czech and English Twitter. In Proceedings of COLING 2014, pages 213–223. our card
  12. R. Giora. 2003. On Our Mind: Salience, Context, and Figurative Language. Oxford University Press. find
  13. H. P. Grice. 1975. Logic and Conversation. In P. Cole and J. L. Morgan, editors, Syntax and Semantics: Vol. 3: Speech Acts, pages 41–58. Academic Press. ↗
  14. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in Twitter. Language resources and evaluation, 47(1):239–268. ↗
  15. Clayton J Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Eighth international AAAI conference on weblogs and social media. find
  16. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proc. of EMNLP 2013, pages 704–714. our card
  17. Aditya Joshi, Pushpak Bhattacharyya, and Mark James Carman. 2017. Automatic sarcasm detection: A survey. ACM Comput. Surv., 50(5):73:1–73:22. ↗ our card
  18. Emilio Sulis, Delia Irazú Hernández Farı́as, Paolo Rosso, Viviana Patti, and Giancarlo Ruffo. 2016. Figurative messages and affect in Twitter: Differences between# irony,# sarcasm and# not. Knowledge-Based Systems, 108:132–143. ↗
  19. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proc. of the 53rd Annual Meeting of the ACL and the 7th Int. Joint Conference on NLP, pages 757–762, Beijing, China. ACL. our card
  20. Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2016a. Exploring the realization of irony in Twitter data. In Proc. of the 10th Int. Conference on Language Resources and Evaluation (LREC 2016). ELRA. our card
  21. Jihen Karoui, Benamara Farah, Véronique Moriceau, Viviana Patti, Cristina Bosco, and Nathalie Aussenac-Gilles. 2017. Exploring the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study. In Proc. of the 15th Conf. of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, volume 1, pages 262–272. our card
  22. Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2016b. Guidelines for annotating irony in social media text, version 2.0. LT3 Technical Report Series. find
  23. Diana Maynard and Mark A. Greenwood. 2014. Who cares about sarcastic tweets? Investigating the impact of sarcasm on sentiment analysis. In Proc. of the 9th Int. Conference on Language Resources and Evaluation, pages 4238–4243. ELRA. our card
  24. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval-2018, New Orleans, LA, USA. ACL. our card
  25. Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3):436–465. ↗
  26. Cynthia Whissell. 2009. Using the revised dictionary of affect in language to quantify the emotional undertones of samples of natural language. Psychological reports, 105(2):509–521. ↗
  27. Saif M Mohammad, Xiaodan Zhu, Svetlana Kiritchenko, and Joel Martin. 2015. Sentiment, emotion, purpose, and style in electoral tweets. Information Processing & Management, 51(4):480–499. ↗
  28. Deirdre Wilson and Dan Sperber. 1992. On Verbal Irony. Lingua, 87(1-2):53–76. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pamungkas-2018-nondicevosulserio-semeval-2018-task-3/