Elamur Research⌕
☰
home / papers / 2015 / ValenTo: Sentiment Analysis of Figurative Language Tweets with Irony and Sarcasm

ValenTo: Sentiment Analysis of Figurative Language Tweets with Irony and Sarcasm

Paper · 2015 · Delia Irazú Hernández Farías, Emilio Sulis, Viviana Patti et al.

Summary

The paper presents the ValenTo team's entry to SemEval-2015 Task 11. The task asks systems to score the sentiment of tweets that contain irony, sarcasm or metaphor on an 11-point scale from −5 to +5. Such figurative devices often flip the literal polarity, and that misleads standard sentiment tools. The authors could still download 7,390 tweets of the roughly 9,000 released. From these they extracted two kinds of features:

  • Structural cues: length, punctuation, POS counts, uppercase, URLs, emoticons, hashtags, mentions, retweets, and a polarity-flip flag for #sarcasm/#not.
  • Lexicon scores: many affective, emotion and psycholinguistic resources, such as AFINN, ANEW, DAL, SenticNet, LIWC and NRC.

Classification worked poorly once the label set grew beyond three classes, so they switched to linear regression in Weka. The submitted system ranked 6th of 15 teams, with CS 0.634 and MSE 2.999. Its results on the ironic and sarcastic subsets came close to the winners'. The work suggests that word-level emotional and psycholinguistic knowledge helps with sarcasm and irony, but not much with metaphor.

Key points

  • Data: 7,390 tweets were recoverable for training plus trial. The polarity distribution leans strongly negative, with a median of −2.3 and a mean of −2.
  • Classification (10-fold CV on 6,928 tweets):
    • With 3 classes, F-measure was about 0.79–0.83 (best: NB at 0.829).
    • With 4 classes, it fell to about 0.46.
    • With all 11 classes, it fell to about 0.30–0.32.
  • Regression: Weka linear regression with feature set ft-conf2 reached cosine similarity of 0.8218, 0.8161 and 0.8199 on three random 70/30 splits.
  • Official per-category CS (ValenTo vs. best):
    • sarcasm: 0.895 vs 0.904
    • irony: 0.901 vs 0.918
    • metaphor: 0.393 vs 0.655
    • other: 0.202 vs 0.612
  • Feature ranking: RELIEF placed NRC emotion words and the AFINN and HL polarity lexicons at the top.

Interesting ideas

  • I1. An explicit polarity-reversal feature fires when a tweet carries a self-labelling irony marker (#sarcasm, #not), so the author's own signal is treated as an operator on literal sentiment rather than as just another token.
  • I2. Several lexicons that measure different things (polarity, valence/arousal/dominance, pleasantness/imagery, discrete emotions) are kept as separate features instead of being merged, so each contributes its own dimension.
  • I3. Moving from coarse classes to fine-grained classes exposed a sharp accuracy drop, which motivated treating intensity as a continuous regression target.
  • I4. Breaking scores down by figurative type showed the method works for irony and sarcasm but fails on metaphor, which overall numbers would hide.
  • I5. A feature-relevance ranking (RELIEF) was used to see which knowledge sources actually drive predictions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The irony module could treat such explicit markers as a negation-like operator over the evaluated frame, recording why the reading was flipped.
  • I2 (world): Entity and state attitudes in the text world could store multi-dimensional affect values drawn from separate lexicons, keeping provenance for explanation.
  • I3 (answer): Graded judgements such as how strongly a character feels could be predicted as a scalar with a stated scale, not as many brittle classes.
  • I4 (learn): Evaluations should report per-phenomenon slices (irony, metaphor, literal) so weaknesses of a rule or model are visible rather than averaged away.
  • I5 (learn): A similar ranking could audit which compiled resources or repair rules matter, helping prune unused knowledge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TurinTurin · Italy27 works · 6,505 citations🎓Universitat Politècnica de ValènciaValencia · Spain21 works · 3,275 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works

🧭 Topics 2 Sentiment & emotion · 608 works Idioms & multiword expressions · 155 works

Cite

Cite as Delia Irazú Hernández Farías, Emilio Sulis, Viviana Patti, Giancarlo Ruffo and Cristina Bosco. 2015. ValenTo: Sentiment Analysis of Figurative Language Tweets with Irony and Sarcasm. Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/s15-2117

@inproceedings{farias2015valento,
  title = {{ValenTo: Sentiment Analysis of Figurative Language Tweets with Irony and Sarcasm}},
  author = {Delia Irazú Hernández Farías, Emilio Sulis, Viviana Patti, Giancarlo Ruffo and Cristina Bosco},
  booktitle = {Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)},
  year = {2015},
  doi = {10.18653/v1/s15-2117},
  url = {https://doi.org/10.18653/v1/s15-2117}
}

References

24 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. Sentiwordnet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10). our card
  2. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano, 2014. Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, chapter Modelling Sarcasm in Twitter, a Novel Approach, pages 50–58. our card
  3. Valerio Basile, Andrea Bolioli, Malvina Nissim, Viviana Patti, and Paolo Rosso. 2014. Overview of the Evalita 2014 SENTIment POLarity Classification Task. In Proceedings of the 4th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA’14), pages 50–57. find
  4. Cristina Bosco, Viviana Patti, and Andrea Bolioli. 2013. Developing corpora for sentiment analysis: The case of irony and senti-tut. IEEE Intelligent Systems, 28(2):55–63. ↗
  5. Margaret Bradley and Peter Lang. 1999. Affective norms for english words (ANEW): Instruction manual and affective ratings. Technical report, Citeseer. find
  6. Konstantin Buschmeier, Philipp Cimiano, and Roman Klinger. 2014. An impact analysis of features in a classification approach to irony detection in product reviews. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 42–49. our card
  7. Erik Cambria, Catherine Havasi, and Amir Hussain. 2012. Senticnet 2: A semantic and affective resource for opinion mining and sentiment analysis. In AAAI FLAIRS Conference, pages 202–207. find
  8. Erick Cambria, B. Schuller, Yunqing Xia, and C. Havasi. 2013. New avenues in opinion mining and sentiment analysis. Intelligent Systems, IEEE, 28(2):15– 21, March. ↗
  9. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10, pages 107–116. our card
  10. A. Ghosh, G. Li, T. Veale, P. Rosso, E. Shutova, A. Reyes, and J. Barnden. 2015. Semeval-2015 task 11: Sentiment analysis of figurative language in twitter. In Proc. Int. Workshop on Semantic Evaluation (SemEval-2015), Co-located with NAACL and *SEM. our card
  11. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT ’11, pages 581–586. our card
  12. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, pages 168–177. find
  13. Bing Liu. 2010. Sentiment analysis and subjectivity. In Handbook of Natural Language Processing. find
  14. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). our card
  15. Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence, 29(3):436–465. ↗
  16. Finn Årup Nielsen. 2011. A new ANEW: evaluation of a word list for sentiment analysis in microblogs. ↗
  17. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(12):1–135. ↗
  18. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Language Resources and Evaluation, 47(1):239– 268. ↗
  19. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the EMNLP: Conference on Empirical Methods in Natural Language Processing, pages 704–714. our card
  20. Marko Robnik-Sikonja and Igor Kononenko. 1997. An adaptation of relief for attribute estimation in regression. In Fourteenth International Conference on Machine Learning, pages 296–304. find
  21. Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. Semeval-2014 task 9: Sentiment analysis in twitter. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 73–80, August. our card
  22. Philip J. Stone and Earl B. Hunt. 1963. A computer approach to content analysis: Studies using the general inquirer system. In Proceedings of the May 21-23, 1963, AFIPS ’63 (Spring), pages 241–256. find
  23. Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. In ECAI, volume 215, pages 765–770. ↗
  24. Cynthia Whissell. 2009. Using the revised dictionary of affect in language to quantify the emotional undertones of samples of natural languages. In Psychological Reports. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/farias-2015-valento-sentiment-analysis-figurative-language/