Elamur Research⌕
☰
home / papers / 2018 / INAOE-UPV at SemEval-2018 Task 3: An Ensemble Approach for Irony Detection in Twitter

INAOE-UPV at SemEval-2018 Task 3: An Ensemble Approach for Irony Detection in Twitter

Paper · 2018 · Delia Irazú Hernández Farías, Fernando Sánchez-Vega, Manuel Montes-y-Gómez et al.

Summary

This paper presents the INAOE-UPV entry to SemEval-2018 Task 3, which asks systems to detect irony in English tweets. Subtask A is a binary ironic/non-ironic decision. Subtask B separates four classes: polarity-contrast verbal irony, other verbal irony, situational irony, and non-irony.

The team combined three word-level classifiers:

  • BOW: binary unigram features fed to an SVM.
  • wEmb: a tweet is represented as the centroid of its Google News CBOW vectors, also classified with an SVM.
  • irO: a new lexicon-based voter. It uses PMI-derived lists of ironic and non-ironic terms built from over 165,000 tweets in existing irony corpora.

They tested two ways of merging these. One routes each tweet by how well the embeddings cover its words; the other is a majority vote. On the official Task A ranking, the coverage-based constrained run reached F1 0.6265 (11th place). The unconstrained majority-vote run placed 2nd among unconstrained systems. Task B proved much harder for everyone.

The work matters because it shows that a simple, transparent lexicon of irony cues can compete with embedding classifiers.

Key points

  • How irO votes:

    • Stopwords are removed first.
    • Each remaining word is compared, by cosine similarity over Google News vectors, with its nearest entry in the ironic list and in the non-ironic list.
    • The word casts a positive vote if it is closer to the ironic list and a negative vote otherwise.
    • The tweet is labelled ironic when the summed votes are ≥ 0.

  • How the lexicon was built:

    • Only terms with PMI > 0 were kept.
    • Terms seen fewer than five times per class were discarded.
    • Extra sources were tweets tagged #irony/#sarcasm collected during the 2016 US election week (8–18 November), plus the task training data.

  • Task A dev results (3-fold CV, F1):

    SystemF1
    BOW0.62
    wEmb0.64
    irO0.63
    Coverage ensemble0.63
    Majority vote0.65

    The coverage ensemble uses wEmb when embeddings cover more than 75% of the tweet's words, and BOW otherwise.

  • Task B dev results (F1): BOW 0.48, irO 0.41, wEmb 0.31; the ensembles scored 0.44 and 0.46, so neither beat BOW.

  • Task B official results: 0.3497 and 0.2148. Both runs used per-class word lists and weighted voting. The U2 run, built from four binary classifiers, placed 3rd among unconstrained systems.

  • Constrained vs. unconstrained:

    • Training and test data share about half their vocabulary.
    • The authors suggest data from the same time window can beat external data, which would explain why the constrained run scored higher.

Interesting ideas

  • I1. Pick which model handles an input according to how much of it that model's resources actually cover, falling back to a sparser model for out-of-vocabulary-heavy text.
  • I2. Let each word cast a signed, inspectable vote based on its nearest neighbour in an ironic versus a non-ironic term list, so every decision can be traced to specific words.
  • I3. Build a class-association lexicon by pooling many heterogeneous labelled corpora, scoring terms with PMI and dropping rare items.
  • I4. The error analysis shows that cue words like "really" or hashtag-heavy posts push sincere text toward an ironic label, and that some irony is unreadable without conversational context.
  • I5. Training data drawn from the same time window as the test set can beat larger external data because the topical vocabulary overlaps.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): sentences dense with hashtags, mentions, emoji or unknown tokens could be routed to the perceptron parser instead of gparse (or vice versa), with the routing reason recorded as part of the explanation.
  • I2 (frames): the irony module in prag could attach such per-lexeme evidence to tree nodes (e.g., the predicate and its modifiers) instead of summing over a flat bag, keeping the verdict explainable.
  • I3 (world): the same recipe could compile cue-word lists (irony markers, evaluative intensifiers) into the knowledge layer as weighted, sourced entries that prag can consult.
  • I4 (learn): sincere sentences containing strong irony cues could serve as negative controls in evaluation, checking that the irony detector relies on frame-level contrast rather than surface markers.
  • I5 (learn): when building silver data, record each source's time frame and domain and measure vocabulary overlap with the target, so out-of-period data is weighted down or isolated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universitat Politècnica de ValènciaValencia · Spain21 works · 3,275 citations🔬National Institute of Astrophysics, Optics and ElectronicsPuebla City · Mexico2 works · 16 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Psychology · 798 works Social Psychology · 129 works

🧭 Topics 2 English · 1,373 works Sentiment & emotion · 608 works

Cite

Cite as Delia Irazú Hernández Farías, Fernando Sánchez-Vega, Manuel Montes-y-Gómez and Paolo Rosso. 2018. INAOE-UPV at SemEval-2018 Task 3: An Ensemble Approach for Irony Detection in Twitter. Proceedings of the 12th International Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1097

@inproceedings{farias2018inaoe,
  title = {{INAOE-UPV at SemEval-2018 Task 3: An Ensemble Approach for Irony Detection in Twitter}},
  author = {Delia Irazú Hernández Farías, Fernando Sánchez-Vega, Manuel Montes-y-Gómez and Paolo Rosso},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1097},
  url = {https://doi.org/10.18653/v1/s18-1097}
}

References

25 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jihen Karoui, Farah Benamara, Veronique Moriceau, Viviana Patti, Cristina Bosco, and Nathalie Aussenac-Gilles. 2017. Exploring the Impact of Pragmatic Phenomena on Irony Detection in Tweets: A Multilingual Corpus Study. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain. our card
  2. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014. Modelling Sarcasm in Twitter, a Novel Approach. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 50–58, Baltimore, Maryland, USA. Association for Computational Linguistics. our card
  3. Peng Liu, Wei Chen, Gaoyan Ou, Tengjiao Wang, Dongqing Yang, and Kai Lei. 2014. Sarcasm Detection in Social Media Based on Imbalanced Classification. In Proceedings of the Web-Age Information Management: 15th International Conference, pages 459–471, Macau, China. Springer International Publishing. find
  4. Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Comput. Linguist., 16(1):22–29. ↗ our card
  5. E. Fersini, F. A. Pozzi, and E. Messina. 2015. Detecting Irony and Sarcasm in Microblogs: The Role of Expressive Signals and Ensemble Classifiers. In 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pages 1–8. find
  6. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and Their Compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, pages 3111–3119. ↗ our card
  7. Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden, and Antonio Reyes. 2015. SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation, pages 470–478, Denver, Colorado. Association for Computational Linguistics. our card
  8. Saif M. Mohammad, Xiaodan Zhu, Svetlana Kiritchenko, and Joel Martin. 2015. Sentiment, Emotion, Purpose, and Style in Electoral Tweets. Information Processing & Management, 51(4):480 – 499. ↗
  9. Aniruddha Ghosh and Tony Veale. 2016. Fracking Sarcasm using Neural Network. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 161–169, San Diego, California. Association for Computational Linguistics. ↗ our card
  10. Debora Nozza, Elisabetta Fersini, and Enza Messina. 2016. Unsupervised Irony Detection: A Probabilistic Model with Word Embeddings. In Proceedings of the 8th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management, pages 68–76. find
  11. Rachel Giora and Ofer Fein. 1999. Irony: Context and Salience. Metaphor and Symbol, 14(4):241–257. ↗
  12. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks. CoRR, abs/1610.08815. ↗ our card
  13. H. P. Grice. 1975. Logic and Conversation. In P. Cole and J. L. Morgan, editors, Syntax and Semantics: Vol. 3: Speech Acts, pages 41–58. Academic Press. ↗
  14. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA Data Mining Software: An Update. SIGKDD Explor. Newsl., 11(1):10–18. ↗
  15. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm Detection on Czech and English Twitter. In Proceedings of the 25th International Conference on Computational Linguistics, pages 213–223, Dublin, Ireland. Association for Computational Linguistics. our card
  16. Delia Irazú Hernández Farı́as, Viviana Patti, and Paolo Rosso. 2016. Irony Detection in Twitter: The Role of Affective Content. ACM Trans. Internet Technol., 16(3):19:1–19:24. find
  17. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm Detection on Twitter: A Behavioral Modeling Approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM ’15, pages 97–106. find
  18. Delia Irazú Hernández Farı́as and Paolo Rosso. 2016. Irony, Sarcasm, and Sentiment Analysis. Chapter 7. In Federico Pozzi, Elisabetta Fersini, Enza Messina, and Bing Liu, editors, Sentiment Analysis in Social Networks, pages 113–127. Morgan Kaufmann. find
  19. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A Multidimensional Approach for Detecting Irony in Twitter. Language Resources and Evaluation, 47(1):239–268. ↗
  20. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic Sarcasm Detection: A Survey. ACM Comput. Surv., 50(5):73:1–73:22. ↗ our card
  21. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as Contrast between a Positive Sentiment and Negative Situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714, Seattle, Washington, USA. Association for Computational Linguistics. our card
  22. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark James Carman. 2016. Are Word Embedding-based Features Useful for Sarcasm Detection? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1006–1011. ↗ our card
  23. G. Salton, A. Wong, and C. S. Yang. 1975. A vector space model for automatic indexing. Commun. ACM, 18(11):613–620. ↗
  24. Emilio Sulis, Delia Irazú Hernández Farı́as, Paolo Rosso, Viviana Patti, and Giancarlo Ruffo. 2016. Figurative Messages and Affect in Twitter: Differences between #irony, #sarcasm and #not. Knowledge-Based Systems, 108:132–143. ↗
  25. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, SemEval-2018, New Orleans, LA, USA. Association for Computational Linguistics. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/farias-2018-inaoe-upv-semeval-2018-task/