Elamur Research⌕
☰
home / papers / 2018 / ValenTO at SemEval-2018 Task 3: Exploring the Role of Affective Content for Detecting Irony in English Tweets

ValenTO at SemEval-2018 Task 3: Exploring the Role of Affective Content for Detecting Irony in English Tweets

Paper · 2018 · Delia Irazú Hernández Farías, Viviana Patti and Paolo Rosso

Summary

The paper describes the ValenTO entry to SemEval-2018 Task 3, which asked systems to detect irony in English tweets (Task A, binary) and to separate types of irony (Task B, four classes). The team reused emotIDM, an earlier irony model built mostly on affect. It draws on many English lexicons that cover sentiment polarity, discrete emotion categories and dimensional emotion scores, plus a small set of structural cues. The authors noticed that links were unevenly spread across the two classes in the training tweets, so they added a binary URL feature. They compared Decision Tree, Random Forest and SVM in constrained and unconstrained settings. The unconstrained runs added many earlier irony and sarcasm corpora. SVM with emotIDM+URL performed best. Officially the system placed 16th in Task A, about 0.1 F1 behind the winner, and 4th among unconstrained runs. For Task B, combining two classifiers gave a modest gain. The authors read the results as evidence that affective information alone carries much of the irony signal.

Key points

  • Task A training data had 1,911 ironic and 1,923 non-ironic tweets. Task B used four labels: non-irony (1923), verbal irony by polarity contrast (1393), other verbal irony (213) and situational irony (328).
  • Affect resources fall into three groups:
    • Sentiment lexicons: Hu&Liu, General Inquirer, EffectWordNet, Subjectivity lexicon, EmoLex, AFINN, SWN, Semantic Orientation lexicon, SenticNet.
    • Emotion categories: EmoLex, EmoSenticNet, SentiSense, LIWC.
    • Dimensional models: ANEW, DAL, SenticNet.
  • 857 of 3,834 training tweets contained a URL, and 592 of those were non-ironic. In 5-fold cross-validation, SVM with emotIDM+URL reached 0.64 (constrained) and 0.81 (unconstrained). Using affect features alone still gave 0.57 and 0.70.
  • Official Task A F1 was 0.6079 (constrained) and 0.6067 (unconstrained), against a top score of 0.7054. In Task B, F1 was 0.3520, ranking 17th of 31.
  • Per-class accuracy in Task B was 79% for non-irony, 45% for polarity-contrast irony, 40% for situational irony and 9.2% for other irony. Typical errors were tweets with no subjective wording, or tweets whose irony depends on outside context or only on the hashtag.

Interesting ideas

  • I1. Affect is modelled as three separate views of the same words: polarity, discrete emotions, and continuous scores such as valence and arousal. Each view catches a different kind of evaluative clash.
  • I2. Before adding a feature, the authors checked how a cheap surface cue (URLs) was distributed over the classes, and found a strong skew worth encoding.
  • I3. Each class was routed to whichever classifier handled it best: the Decision Tree for the rare irony types, SVM for the rest.
  • I4. Results were broken down by collection hashtag (#irony, #sarcasm, #not) to confirm that the model was not leaning on one source. #not was handled both as an irony marker and as an ordinary negation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The irony module in prag could attach polarity, emotion and valence/arousal values to the predicate and to the described situation, then flag explicit mismatches between them as an explainable irony cue.
  • I2 (learn): Silver-data evaluation could routinely compare cue-by-class distributions to find shortcut signals, and then decide whether to exploit them or to control for them with negative controls.
  • I3 (learn): When several witness models disagree, trust could be assigned per label instead of globally, so rare categories follow the witness that is reliable for them.
  • I4 (tokens): The tokenizer could keep hashtag words available as ordinary lexical tokens, so that #not inside a sentence still parses as negation while a trailing #not is passed on to the pragmatics layer as a marker.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TurinTurin · Italy27 works · 6,505 citations🎓Universitat Politècnica de ValènciaValencia · Spain21 works · 3,275 citations🔬National Institute of Astrophysics, Optics and ElectronicsPuebla City · Mexico2 works · 16 citations

Also on the byline: National Institute for Astrophysics.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksSentiment & emotion · 608 works

Cite

Cite as Delia Irazú Hernández Farías, Viviana Patti and Paolo Rosso. 2018. ValenTO at SemEval-2018 Task 3: Exploring the Role of Affective Content for Detecting Irony in English Tweets. Proceedings of the 12th International Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1105

@inproceedings{farias2018valento,
  title = {{ValenTO at SemEval-2018 Task 3: Exploring the Role of Affective Content for Detecting Irony in English Tweets}},
  author = {Delia Irazú Hernández Farías, Viviana Patti and Paolo Rosso},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1105},
  url = {https://doi.org/10.18653/v1/s18-1105}
}

References

37 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Delia Irazú Hernández Farı́as, Viviana Patti, and Paolo Rosso. 2016. Irony Detection in Twitter: The Role of Affective Content. ACM Trans. Internet Technol., 16(3):19:1–19:24. find
  2. Laura Alba-Juez and Salvatore Attardo. 2014. The Evaluative Palette of Verbal Irony. In Geoff Thompson and Laura Alba-Juez, editors, Evaluation in Context, pages 93–116. John Benjamins Publishing Company, Amsterdam/Philadelphia. ↗
  3. Delia Irazú Hernández Farı́as and Paolo Rosso. 2016. Irony, Sarcasm, and Sentiment Analysis. Chapter 7. In Federico A. Pozzi, Elisabetta Fersini, Enza Messina, and Bing Liu, editors, Sentiment Analysis in Social Networks, pages 113–127. Morgan Kaufmann. find
  4. Jorge Carrillo de Albornoz, Laura Plaza, and Pablo Gervás. 2012. SentiSense: An Easily Scalable Concept-based Affective Lexicon for Sentiment Analysis. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 3562–3567. European Language Resources Association (ELRA). find
  5. Irazú Hernández Farı́as, José-Miguel Benedı́, and Paolo Rosso. 2015. Applying Basic Features from Sentiment Analysis for Automatic Irony Detection. In Pattern Recognition and Image Analysis, volume 9117 of Lecture Notes in Computer Science, pages 337–344. Springer International Publishing. find
  6. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014. Modelling Sarcasm in Twitter, a Novel Approach. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 50–58. Association for Computational Linguistics.
  7. Irazú Hernández-Farias, Davide Buscaldi, and Belém Priego-Sánchez. 2014. IRADABE: Adapting English Lexicons to the Italian Sentiment Polarity Classification Task. In Proceedings of the First Italian Conference on Computational Linguistics (CLiC-it 2014) & the Fourth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian EVALITA 2014, pages 75–81. find
  8. Valerio Basile, Andrea Bolioli, Malvina Nissim, Viviana Patti, and Paolo Rosso. 2014. Overview of the Evalita 2014 SENTIment POLarity classification task. In Proceedings of the Fourth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian EVALITA 2014, pages 50–57. find
  9. Margaret M Bradley and Peter J Lang. 1999. Affective Norms for English Words (ANEW): Instruction Manual and Affective Ratings. Technical report, Center for Research in Psychophysiology, University of Florida, Gainesville, Florida. find
  10. Minqing Hu and Bing Liu. 2004. Mining and Summarizing Customer Reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, pages 168–177, Seattle, WA, USA. ACM. find
  11. Konstantin Buschmeier, Philipp Cimiano, and Roman Klinger. 2014. An Impact Analysis of Features in a Classification Approach to Irony Detection in Product Reviews. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 42–49.
  12. Jihen Karoui, Farah Benamara, Véronique Moriceau, Nathalie Aussenac-Gilles, and Lamia Hadrich-Belguith. 2015. Towards a Contextual Pragmatic Model to Detect Irony in Tweets. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 644–650. Association for Computational Linguistics.
  13. Erik Cambria, Daniel Olsher, and Dheeraj Rajagopal. 2014. SenticNet 3: A Common and Common-Sense Knowledge Base for Cognition-Driven Sentiment Analysis. In Proceedings of AAAI Conference on Artificial Intelligence, pages 1515–1521. AAAI. find
  14. Jihen Karoui, Farah Benamara, Veronique Moriceau, Viviana Patti, Cristina Bosco, and Nathalie Aussenac-Gilles. 2017. . In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, Valencia, Spain. Association for Computational Linguistics. find
  15. Yoonjung Choi and Janyce Wiebe. 2014. +/-EffectWordNet: Sense-level Lexicon Acquisition for Opinion Inference. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1181–1191. find
  16. Philip J. Stone and Earl B. Hunt. 1963. A Computer Approach to Content Analysis: Studies Using the General Inquirer System. In Proceedings of the May 21-23, 1963, Spring Joint Computer Conference, AFIPS ’63 (Spring), pages 241–256. ACM. find
  17. Joan Lucariello. 2014. Situational Irony: A Concept of Events Gone Awry. Journal of Experimental Psychology: General, 123(2):129–145. find
  18. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a Word–Emotion Association Lexicon. Computational Intelligence, 29(3):436–465. ↗
  19. Emilio Sulis, Delia Irazú Hernández Farı́as, Paolo Rosso, Viviana Patti, and Giancarlo Ruffo. 2016. Figurative Messages and Affect in Twitter: Differences between #irony, #sarcasm and #not. Knowledge-Based Systems, 108:132–143. ↗
  20. Saif M. Mohammad, Xiaodan Zhu, Svetlana Kiritchenko, and Joel Martin. 2015. Sentiment, Emotion, Purpose, and Style in Electoral Tweets. Information Processing & Management, 51(4):480 – 499. ↗
  21. Maite Taboada and Jack Grieve. 2004. Analyzing Appraisal Automatically. In Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications, pages 158–161, Stanford, US. AAAI. find
  22. Malvina Nissim and Viviana Patti. 2016. Semantic aspects in sentiment analysis. In Fersini Elisabetta, Bing Liu, Enza Messina, and Federico Pozzi, editors, Sentiment Analysis in Social Networks, chapter 3, pages 31–48. Elsevier. ↗
  23. Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2016. Exploring the realization of irony in Twitter data. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). European Language Resources Association (ELRA).
  24. Debora Nozza, Elisabetta Fersini, and Enza Messina. 2016. Unsupervised Irony Detection: A Probabilistic Model with Word Embeddings. In Proceedings of the 8th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management, pages 68–76. find
  25. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval-2018, New Orleans, LA, USA. Association for Computational Linguistics.
  26. James W. Pennebaker, Martha E. Francis, and Roger J. Booth. 2001. Linguistic Inquiry and Word Count: LIWC 2001. Mahway: Lawrence Erlbaum Associates, 71. find
  27. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks. CoRR, abs/1610.08815. ↗
  28. Byron C. Wallace, Do Kook Choe, and Eugene Charniak. 2015. Sparse, Contextually Informed Models for Irony Detection: Exploiting User Communities, Entities and Sentiment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1035–1044. Association for Computational Linguistics.
  29. Soujanya Poria, Alexander Gelbukh, Amir Hussain, Newton Howard, Dipankar Das, and Sivaji Bandyopadhyay. 2013. Enhanced SenticNet with Affective Labels for Concept-Based Opinion Mining. IEEE Intelligent Systems, 28(2):31–38. ↗
  30. Angela P. Wang. 2013. #irony or #sarcasm — A Quantitative and Qualitative Study Based on Twitter. In Proceedings of the 27th Pacific Asia Conference on Language, Information, and Computation, pages 349–356. National Chengchi University.
  31. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm Detection on Czech and English Twitter. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics, pages 213–223. Dublin City University and Association for Computational Linguistics.
  32. Cynthia Whissell. 2009. Using the Revised Dictionary of Affect in Language to Quantify the Emotional Undertones of Samples of Natural Languages. Psychological Reports, 2(105):509–521. find
  33. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A Multidimensional Approach for Detecting Irony in Twitter. Language Resources and Evaluation, 47(1):239–268. ↗
  34. Deirdre Wilson and Dan Sperber. 1992. On Verbal Irony. Lingua, 87(1-2):53–76. ↗
  35. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as Contrast between a Positive Sentiment and Negative Situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714. Association for Computational Linguistics.
  36. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing Contextual Polarity in Phrase-level Sentiment Analysis. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, pages 347–354. Association for Computational Linguistics.
  37. Dan Sperber and Deirdre Wilson. 1986. Relevance: Communication and Cognition. Harvard University Press, Cambridge, MA, USA. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/farias-2018-valento-semeval-2018-task-3/