Elamur Research⌕
☰
home / papers / 2018 / NIHRIO at SemEval-2018 Task 3: A Simple and Accurate Neural Network Model for Irony Detection in Twitter

NIHRIO at SemEval-2018 Task 3: A Simple and Accurate Neural Network Model for Irony Detection in Twitter

Paper · 2018 · Thanh Vu, Dat Quoc Nguyen, Xuan-Son Vu et al.

Summary

This SemEval-2018 Task 3 system detects irony in English tweets in two settings: a yes/no decision (Subtask A) and a four-way choice among non-irony, polarity-contrast verbal irony, other verbal irony and situational irony (Subtask B). The authors deliberately train on the version of the data with explicit irony hashtags (#not, #sarcasm, #irony) removed, so training matches the test set. Tweets are cleaned first: elongated spellings are normalized with a dictionary, user mentions and links become placeholder tokens, and emoji are rewritten as text. Each tweet is then turned into a hand-engineered vector of lexical, syntactic, semantic and polarity features. This vector feeds a small Multilayer Perceptron with two ReLU hidden layers. Ten such networks, trained across cross-validation folds, vote on every test tweet. The system placed third by accuracy and fifth by F1 in both subtasks. It outperformed LSTM, CNN, SVM and other classifiers the team tried. The work shows that a carefully featured shallow network can match more complex models on small, noisy data.

Key points

  • The data has 4,618 tweets (2,222 ironic, 2,396 non-ironic), split into 3,834 for training and 784 for testing. Situational irony is rare, with only 205 training examples.
  • The input vector has 2,759 features:
    • 2,002 lexical: top-1,000 tf-idf word and character 1–3-grams, plus length counts;
    • 45 syntactic: NLTK POS tags weighted by tf-idf;
    • 700 semantic: averaged 300-d GloVe vectors, a 100-d LSI vector, and Brown cluster counts at 80, 100 and 120 clusters;
    • 12 polarity: Hu and Liu lexicon words, emoji polarity and negation flags.
  • The MLP hidden layers are (800, 400) for Subtask A and (800, 300) for Subtask B. Training used Tensorflow with cross-entropy loss and L2 regularization, and the final label comes from a 10-model fold-based vote.
  • In Subtask A (44 teams), the system reached accuracy 65.94 (3rd) and F1 64.76 (5th). In Subtask B (32 teams), it reached accuracy 70.15 (3rd) and macro F1 44.37 (5th).
  • Per-class F1 in Subtask B was 60.73 for polarity-contrast irony, 31.75 for other verbal irony and only 8.70 for situational irony. An LSTM scored about 92% versus the MLP's 87% when hashtags were kept, but fell behind the MLP once they were removed.

Interesting ideas

  • I1. Polarity-contrast irony is captured by simply counting positive and negative cues, both words and emoji, together with a negation flag, so that a clash between sentiments becomes visible.
  • I2. Removing the giveaway irony hashtags from training changes which model wins: the LSTM leads with them but loses without them. This shows how a single surface marker can hide what a model has actually learned.
  • I3. A fold-based ensemble lets each of ten models tune on its own held-out fold, and their majority vote gives the final label. This is a cheap way to stabilize results on small data.
  • I4. The per-class breakdown exposes a near-total failure on situational irony, which aggregate scores conceal and the authors tie to having only 205 examples.
  • I5. Before any analysis, the system normalizes elongated words with a lexicon, maps mentions and URLs to placeholders, and verbalizes emoji.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): the irony module in prag could flag clauses where a positive evaluative predicate meets a negative state of the described situation, using the frame structure rather than raw counts.
  • I2 (learn): build negative-control test sets where obvious cue words are stripped, and check that the irony or speech-act decisions still hold.
  • I3 (learn): when training small classifiers or repair rules on scarce silver data, fold-wise models plus a vote can reduce variance without extra annotation.
  • I4 (learn): always report per-label results for rare frame or speech-act types, so that data starvation is caught early.
  • I5 (tokens): the en tokenizer could apply the same dictionary normalization and placeholder typing for social-media text, which would give the tagger and parser cleaner input.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia60 works · 21,199 citations🎓Newcastle UniversityNewcastle upon Tyne · United Kingdom8 works · 588 citations🎓Deakin UniversityBurwood · Australia2 works · 973 citations🎓Umeå UniversityUmeå · Sweden2 works · 37 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksSentiment & emotion · 608 works

Cite

Cite as Thanh Vu, Dat Quoc Nguyen, Xuan-Son Vu, Dai Quoc Nguyen, Michael Catt and Michael Trenell. 2018. NIHRIO at SemEval-2018 Task 3: A Simple and Accurate Neural Network Model for Irony Detection in Twitter. Proceedings of the 12th International Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s18-1085 · arXiv 1804.00520

@inproceedings{vu2018nihrio,
  title = {{NIHRIO at SemEval-2018 Task 3: A Simple and Accurate Neural Network Model for Irony Detection in Twitter}},
  author = {Thanh Vu, Dat Quoc Nguyen, Xuan-Son Vu, Dai Quoc Nguyen, Michael Catt and Michael Trenell},
  booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation},
  year = {2018},
  doi = {10.18653/v1/s18-1085},
  eprint = {1804.00520},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/s18-1085}
}

References

38 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. find
  2. Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a #twitter. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, pages 368–378.
  3. Bo Han, Paul Cook, and Timothy Baldwin. 2012. Automatically constructing a normalisation dictionary for microblogs. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 421–432. find
  4. Frank E Harrell. 2001. Ordinal logistic regression. In Regression modeling strategies, pages 331–343. ↗
  5. Apoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow, and Rebecca Passonneau. 2011. Sentiment analysis of twitter data. In Proceedings of the workshop on languages in social media, pages 30–38.
  6. Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. 1998. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18–28. ↗
  7. Francesco Barbieri and Horacio Saggion. 2014. Modelling irony in twitter. In Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 56–64.
  8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735– 1780. ↗
  9. Eli Bartov, Lucile Faurel, and Partha Mohanram. 2015. Can twitter help predict firm-level earnings and stock returns? The Accounting Review. ↗
  10. K. Hornik, M. Stinchcombe, and H. White. 1989. Multilayer feedforward networks are universal approximators. Neural Netw., 2(5):359–366. ↗
  11. Johan Bollen, Huina Mao, and Xiaojun Zeng. 2011. Twitter mood predicts the stock market. Journal of computational science, 2(1):1–8. ↗
  12. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177. find
  13. Cristina Bosco, Viviana Patti, and Andrea Bolioli. 2013. Developing corpora for sentiment analysis: The case of irony and senti-tut. IEEE Intelligent Systems, 28(2):55–63. ↗
  14. Yu-Hsiang Huang, Hen-Hsen Huang, and Hsin-Hsi Chen. 2017. Irony detection with attentive recurrent neural networks. In European Conference on Information Retrieval, pages 534–540. find
  15. Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Class-based n-gram models of natural language. Comput. Linguist., 18(4):467–479. find
  16. Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys, 50(5):73. ↗
  17. Erik Cambria, Soujanya Poria, Devamanyu Hazarika, and Kenneth Kwok. 2018. Senticnet 5: discovering conceptual primitives for sentiment analysis by means of context embeddings. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. find
  18. Sandeepa Kannangara. 2018. Mining twitter for fine-grained political opinion polarity classification, ideology detection and sarcasm detection. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 751–752. find
  19. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 107–116.
  20. Jihen Karoui, Benamara Farah, Véronique Moriceau, Viviana Patti, Cristina Bosco, and Nathalie Aussenac-Gilles. 2017. Exploring the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study. In Proceedings of the ACM International Conference on Web Search and Data Mining, pages 97–106. 15th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 262–272.
  21. Kumar Ravi and Vadlamani Ravi. 2017. A novel automatic satire and irony detection using ensembled feature selection and data mining. Knowledge-Based Systems, 120:15–33. ↗
  22. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1746–1751.
  23. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Lang. Resour. Eval., 47(1):239–268. ↗
  24. Efthymios Kouloumpis, Theresa Wilson, and Johanna D Moore. 2011. Twitter sentiment analysis: The good the bad and the omg! Proceedings of the 5th International Conference on Web and Social Media, pages 538–541. find
  25. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation, pages 502–518.
  26. Saskia Le Cessie and Johannes C Van Houwelingen. 1992. Ridge estimators in logistic regression. Applied statistics, pages 191–201. ↗
  27. Percy Liang. 2005. Semi-supervised learning for natural language. Ph.D. thesis, Massachusetts Institute of Technology. find
  28. Emilio Sulis, Delia Irazú Hernández Farı́as, Paolo Rosso, Viviana Patti, and Giancarlo Ruffo. 2016. Figurative messages and affect in twitter: Differences between# irony,# sarcasm and# not. Knowledge-Based Systems, 108:132–143. ↗
  29. Kun-Lin Liu, Wu-Jun Li, and Minyi Guo. 2012. Emoticon smoothed language models for twitter sentiment analysis. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, pages 1678–1684. find
  30. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, page to appear.
  31. Michael Nofer and Oliver Hinz. 2015. Using twitter to predict the stock market. Business & Information Systems Engineering, 57(4):229–242. ↗
  32. Tien Thanh Vu, Shu Chang, Quang Thuy Ha, and Nigel Collier. 2012. An experiment in integrating sentiment features for tech stock prediction in twitter. In The COLING Workshop on Information Extraction and Entity Analytics on Social Media Data, pages 23–38. find
  33. Debora Nozza, Elisabetta Fersini, and Enza Messina. 2017. A multi-view sentiment corpus. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 273–280. find
  34. Nuno Oliveira, Paulo Cortez, and Nelson Areal. 2017. The impact of microblogging data for stock market prediction: using twitter to predict returns, volatility, trading volume and survey sentiment indices. Expert Systems with Applications, 73:125–144. ↗
  35. Alexander Pak and Patrick Paroubek. 2010. Twitter as a corpus for sentiment analysis and opinion mining. In Proceedings of the Seventh conference on International Language Resources and Evaluation.
  36. Christos H. Papadimitriou, Hisao Tamaki, Prabhakar Raghavan, and Santosh Vempala. 1998. Latent semantic indexing: A probabilistic analysis. In Proceedings of the Seventeenth ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, pages 159–168. find
  37. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532–1543.
  38. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vu-2018-nihrio-semeval-2018-task-3/