Elamur Research⌕
☰
home / papers / 2014 / An Impact Analysis of Features in a Classification Approach to Irony Detection in Product Reviews

An Impact Analysis of Features in a Classification Approach to Irony Detection in Product Reviews

Summary

Sentiment analysis of product reviews can be misled by ironic or sarcastic texts, where the literal polarity differs from what the author means. Filatova (2012) released an annotated Amazon review corpus for irony, but no classification results had been reported on it. Buschmeier, Cimiano and Klinger fill this gap by treating irony detection as binary supervised classification of whole reviews. They combine a bag-of-words representation with 29 hand-crafted cues taken from earlier studies. These include a mismatch between star rating and word polarity, hyperbole, punctuation patterns, ellipses, interjections, laughter markers and emoticons. They compare five standard classifiers and run an ablation study, removing features one at a time to measure each contribution. Logistic regression with all features performs best, at an F1 of about 74%. However, much of the signal comes from the star-rating metadata. Specific cues alone reach only about 51%, and removing the rating-based Imbalance feature clearly lowers performance. The work provides a reproducible baseline and shows that irony detection from text alone remains an open problem.

Key points

  • Data: Filatova's (2012) Amazon review corpus, annotated through Amazon Mechanical Turk, with 437 ironic and 817 non-ironic reviews.
  • Features: The full set has 21,773 features, made up of bag-of-words plus 29 task-specific ones. Word polarity comes from the Hu and Liu (2004) lexicon of about 6,800 words.
  • Classifiers: SVM (linear kernel, libSVM), multinomial naïve Bayes, logistic regression, decision tree and random forest, all run in scikit-learn. Code is available at github.com/kbuschme/irony-detection.
  • Results:
    • Logistic regression with all features reaches F1 ≈ 74%.
    • Task-specific features alone peak at 50.9% F1, with high precision but very low recall.
    • Bag-of-words combined with specific features but without rating information reaches about 68%.
    • A baseline using only the star rating is highly competitive, but it is biased and requires metadata.
  • Most useful cues: Imbalance, the Positive feature set, Quotes and Pos/Neg&Ellipsis. The authors suggest future work on out-of-domain vocabulary, semantic similarity, and transfer across text sources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

28 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 16 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-2608

@inproceedings{buschmeier2014impact,
  title = {{An Impact Analysis of Features in a Classification Approach to Irony Detection in Product Reviews}},
  author = {Buschmeier, Konstantin and Cimiano, Philipp and Klinger, Roman},
  booktitle = {Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis},
  year = {2014},
  doi = {10.3115/v1/w14-2608},
  url = {https://doi.org/10.3115/v1/w14-2608}
}

References

38 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Meyer Howard Abrams. 1957. A Glossary of Literary Terms. Cengage Learning Emea, 9th edition. find
  2. Salvatore Attardo. 2000. Irony markers and functions: Towards a goal-oriented theory of irony and its processing. Rask: Internationalt Tidsskrift for Sprog og Kommunikation, 12:3–20. find
  3. Leo Breiman, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone. 1984. Classification and Regression Trees. Wadsworth, Belmont, California. find
  4. Leo Breiman. 2001. Random forests. Machine Learning, 45(1):5–32. find
  5. British Dictionary. 2014. MacMillan Publishers. Online: http://www.macmillandictionary.com/dictionary/british/irony. accessed April 28, 2014. ↗
  6. Paula Carvalho, Luı́s Sarmento, Mário J. Silva, and Eugénio de Oliveira. 2009. Clues for detecting irony in user-generated contents: oh. . . !! it’s “so easy” ;-). In Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, TSA ’09, pages 53–56, New York, NY, USA. ACM. find
  7. Chih-Chung Chang and Chih-Jen Lin. 2011. Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3). find
  8. Herbert H. Clark and Richard J. Gerrig. 1984. On the pretense theory of irony. Journal of Experimental Psychology: General, 113(1):121–126. ↗
  9. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning, 20(3):273– 297. ↗
  10. Marlena A. Creusere. 2007. A developmental test of theoretical perspective on the understanding of verbal irony: Children’s recognition of allusion and pragmatic insincerity. In Raymond W. Jr. Gibbs and Herbert L. Colston, editors, Irony in Language and Thought: A Cognitive Science Reader, chapter 18, pages 409–424. Lawrence Erlbaum Associates, 1st edition. find
  11. Dmitry Davidov and Ari Rappoport. 2006. Efficient unsupervised discovery of word categories using symmetric patterns and high frequency words. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 297–304, Sydney, Australia, July. Association for Computational Linguistics. find
  12. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10, pages 107–116, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  13. Duden. 2014. Duden Verlag. Online: http://www.duden.de/rechtschreibung/Ironie. accessed April 28, 2014. ↗
  14. Rong-En Fan, Pai-Hsuen Chen, and Chih-Jen Lin. 2005. Working set selection using second order information for training support vector machines. Jounral of Machine Learning Reasearch, 6:1889– 1918. ↗
  15. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 392–398, Istanbul, Turkey, May. European Language Resources Association (ELRA). find our card
  16. Raymond W. Jr. Gibbs. 2007. Irony in talk among friends. In Raymond W. Jr. Gibbs and Herbert L. Colston, editors, Irony in Language and Thought: A Cognitive Science Reader, chapter 15, pages 339–360. Lawrence Erlbaum Associates, 1st edition, May. find
  17. Melanie Harris Glenwright and Penny M. Pexman. 2007. Children’s perceptions of the social functions of verbal irony. In Raymond W. Jr. Gibbs and Herbert L. Colston, editors, Irony in Language and Thought: A Cognitive Science Reader, chapter 20, pages 447–464. Lawrence Erlbaum Associates, 1st edition. find
  18. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2, HLT ’11, pages 581–586, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  19. Trevor Hastie, Robert Tibshirani, and Jerome H. Friedman. 2009. Elements of Statistical Learning. Springer, 2nd edition. find
  20. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’04, pages 168–177, New York, NY, USA. ACM. find
  21. Roger J. Kreuz and Gina M. Caucci. 2007. Lexical influences on the perception of sarcasm. In Proceedings of the Workshop on Computational Approaches to Figurative Language, pages 1–4, Rochester, New York, April. Association for Computational Linguistics. find our card
  22. Roger J. Kreuz. 1996. The use of verbal irony: Cues and constraints. In Jeffery S. Mio and Albert N. Katz, editors, Metaphor: Implications and Applications, pages 23–38, Mahwah, NJ, October. Lawrence Erlbaum Associates. find
  23. Sachi Kumon-Nakamura, Sam Glucksberg, and Mary Brown. 1995. How about another piece of pie: The allusional pretense theory of discourse irony. Journal of Experimental Psychology: General, 124(1):3– 21, Mar 01. Last updated - 2013-02-23. ↗
  24. Lillian Lee. 2009. A tempest or, on the flood of interest in: sentiment analysis, opinion mining, and the computational treatment of subjective language. Tutorial at ICWSM, May. find
  25. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press. ↗
  26. Merriam Webster Dictionary. 2014. Merriam Webster Inc. Online: www.merriam-webster.com/dictionary/irony. accessed April 28, 2014. find
  27. Constantine Nakassis and Jesse Snedeker. 2002. Beyond sarcasm: Intonation and context as relational cues in children’s recognition of irony. In A. Greenhill, M. Hughs, H. Littlefield, and H. Walsh, editors, Proceedings of the Twenty-sixth Boston University Conference on Language Development, Somerville, MA, July. Cascadilla Press. find
  28. New Oxford American Dictionary. 2014. Oxford University Press. Online: http://www.oxforddictionaries.com/us/definition/american_english/ironic. accessed April 28, 2014. ↗
  29. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  30. Antonio Reyes, Paolo Rosso, and Davide Buscaldi. 2012. From humor recognition to irony detection: The figurative language of social media. Data & Knowledge Engineering, 74:1–12. ↗
  31. Patricia Rockwell. 2005. Sarcasm on television talk shows: Determining speaker intent through verbal and nonverbal cues. In Anita V. Clark, editor, Psychology of Moods, chapter 6, pages 109–122. Nova Science Pubishers Inc. find
  32. Joseph Tepperman, David Traum, and Shrikanth S. Narayanan. 2006. “yeah right”: Sarcasm recognition for spoken dialogue systems. In Proceedings of InterSpeech, pages 1838–1841, Pittsburgh, PA, September. find
  33. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. ICWSM – A Great Catchy Name: Semi-Supervised Recognition of Sarcastic Sentences in Online Product Reviews. In Proceedings of the Fourth International AAAI Conference on Weblogs and Social Media, pages 162–169. The AAAI Press. find
  34. Akira Utsumi. 1996. A unified theory of irony and its computational formalization. In Proceedings of the 16th conference on Computational linguistics - Volume 2, COLING ’96, pages 962–967, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  35. Deirdre Wilson and Dan Sperber. 1992. On verbal irony. Lingua, 87:53–76. ↗
  36. Deirdre Wilson and Dan Sperber, 2012. Explaining Irony, chapter 6, pages 123–145. Cambridge University Press, 1st edition, April. Hsiang-Fu. find
  37. Yu, Fang-Lan Huang, and Chih-Jen Lin. 2011. Dual coordinate descent methods for logistic regression and maximum entropy. Machine Learning, 85(1–2):41–75, October. find
  38. Harry Zhang. 2004. The optimality of naive bayes. In Valerie Barr and Zdravko Markov, editors, Proceedings of the Seventeenth International Florida Artificial Intelligence Research Society (FLAIRS) Conference, pages 3–9. AAAI Press. find

↑ back to top