Elamur Research⌕
☰
home / papers / 2017 / Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation

Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation

Summary

The paper introduces a new NLP task, sarcasm interpretation: given a sarcastic text, produce a plain, non-sarcastic version that carries the speaker's real message. The authors argue this helps sentiment analysis and opinion mining systems, which sarcasm misleads, and could help people who find figurative language hard to understand. They built the first parallel corpus for the task. It contains 3000 English tweets tagged #sarcasm, and hired Fiverr workers wrote five honest rewrites of each tweet. They treat the task as monolingual machine translation and compare Moses with an RNN encoder-decoder. They then propose SIGN, an MT-based method that focuses on sentiment-bearing words, groups them into clusters, and replaces them during translation. On n-gram metrics, all systems score about the same. Human judges, however, rate SIGN's outputs clearly higher for adequacy and sentiment polarity. The paper also shows that standard automatic metrics barely agree with human judgments, so the task needs new evaluation methods.

Key points

  • Dataset: 3000 text-only sarcastic tweets posted January–June 2016, each with five human interpretations. The authors removed retweets, non-English tweets, and tweets with URLs or images. The vocabulary has 8788 unique words. The data, annotation guidelines, and code are released at github.com/Lotemp/SarcasmSIGN.
  • Evaluation:
    • Automatic metrics: BLEU, ROUGE, PINC, and PINC combined with a sigmoid of BLEU.
    • Human ratings: fluency, adequacy, and whether the output captures the intended sentiment.
  • Correlation with humans: Pearson correlations between automatic and human scores are weak, at most 0.12 for fluency, 0.13–0.18 for sentiment, and 0.19–0.24 for adequacy. The fluency correlations are not statistically significant.
  • Why Moses falls short: Moses tunes toward BLEU, so it tends to copy the tweet unchanged. SIGN actively rewrites sentiment words and turns "fun" into "boring". A comparison with the SIGN-oracle variant suggests that better clustering of sentiment words matters more than better context selection.
  • Remaining failure cases:
    • tweets that need world knowledge, such as the one about LeBron;
    • tweets with no explicit sentiment word;
    • sentiment carried by phrases or slang, such as "can't wait" and "totally".

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cited by

24 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 14 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1155 · arXiv 1704.06836

@inproceedings{peled2017sarcasm,
  title = {{Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation}},
  author = {Peled, Lotem and Reichart, Roi},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1155},
  eprint = {1704.06836},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p17-1155}
}

References

40 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. In Ninth International AAAI Conference on Web and Social Media. http://dblp.uni-trier.de/rec/bib/conf/icwsm/BammanS15. ↗
  2. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 597–604. www.aclweb.org/anthology/P05-1074. ↗ our card
  3. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014. Proceedings of the 5th workshop on computational approaches to subjectivity, sentiment and social media analysis. pages 50–58. https://doi.org/10.3115/v1/W14-2609. ↗ our card
  4. David L Chen and William B Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, pages 190–200. www.aclweb.org/anthology/P11-1020. ↗ our card
  5. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proc. of EMNLP. https://doi.org/10.3115/v1/d14-1179. ↗
  6. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the fourteenth conference on computational natural language learning. Association for Computational Linguistics, pages 107–116. https://www.aclweb.org/anthology/W/W10/W10-2914.pdf. ↗ our card
  7. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of the Sixth Workshop on Statistical Machine Translation. Association for Computational Linguistics, pages 85–91. www.aclweb.org/anthology/W11-2107. ↗ our card
  8. George Doddington. 2002. Automatic evaluation of machine translation quality using ngram co-occurrence statistics. In Proceedings of the second international conference on Human Language Technology Research. Morgan Kaufmann Publishers Inc., pages 138–145. https://doi.org/10.3115/1289189.1289273. ↗
  9. Andrea Esuli and Fabrizio Sebastiani. 2006. Sentiwordnet: A publicly available lexical resource for opinion mining. In Proceedings of LREC. http://aclweb.org/anthology/L06-1225. ↗ our card
  10. Raymond W Gibbs and Herbert L Colston. 2007. Irony in language and thought: A cognitive science reader. Psychology Press. ↗
  11. Laurence Gillick and Stephen J Cox. 1989. Some statistical issues in the comparison of speech recognition algorithms. In Proc. of ICASSP. IEEE. https://doi.org/10.1109/ICASSP.1989.266481. ↗
  12. Roberto González-Ibánez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2. Association for Computational Linguistics, pages 581–586. http://www.aclweb.org/anthology/P11-2102. ↗ our card
  13. Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation. Association for Computational Linguistics, pages 187– 197. www.aclweb.org/anthology/W11-2123. ↗ our card
  14. Andreas M Kaplan and Michael Haenlein. 2011. Two hearts in three-quarter time: How to waltz the social media/viral marketing dance. Business Horizons 54(3):253–263. https://doi.org/10.1016/j.bushor.2011.01.006. ↗
  15. Yael Kimhi. 2014. Theory of mind abilities and deficits in autism spectrum disorders. Topics in Language Disorders 34(4):329–343. https://doi.org/10.1097/tld.0000000000000033. ↗
  16. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions. Association for Computational Linguistics, pages 177– 180. https://www.aclweb.org/anthology/P/P07/P07-2.pdf. ↗
  17. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1. Association for Computational Linguistics, pages 48–54. www.aclweb.org/anthology/N/N03/N03-1017.ps. ↗
  18. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, pages 302–308. https://doi.org/10.3115/v1/P14-2050. ↗
  19. Chin-Yew Lin. 2004. Text summarization branches out (acl-04 workshop). http://aclweb.org/anthology/W04-1013. ↗
  20. Diana Maynard, Kalina Bontcheva, and Dominic Rout. 2012. Challenges in developing opinion mining tools for social media. Proceedings of the@ NLP can u tag# usergeneratedcontent (LREC-12 workshop) pages 15–22. find
  21. Diana Maynard and Mark A Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In LREC. pages 4238–4243. http://dblp.uni-trier.de/rec/bib/conf/lrec/MaynardG14. ↗ our card
  22. Inc Merriam-Webster. 1983. Webster’s ninth new collegiate dictionary. Merriam-Webster. https://doi.org/10.1353/dic.1984.0017. ↗
  23. George A Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J Miller. 1990. Introduction to wordnet: An on-line lexical database. International journal of lexicography 3(4):235–244. https://doi.org/10.1093/ijl/3.4.235. ↗
  24. Douglas Colin Muecke. 1982. Irony and the Ironic. Methuen. find
  25. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational linguistics 29(1):19–51. http://aclweb.org/anthology/J03-1002. ↗
  26. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and trends in information retrieval 2(1-2):1–135. http://dblp.uni-trier.de/rec/bib/journals/ftir/PangL07. ↗
  27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 311–318. www.aclweb.org/anthology/P02-1040.pdf. ↗ our card
  28. Candida C Peterson, Henry M Wellman, and Virginia Slaughter. 2012. The mind behind the message: Advancing theory-of-mind scales for typically developing children, and those with deafness, autism, or asperger syndrome. Child development 83(2):469–485. https://doi.org/10.1111/j.1467-8624.2011.01728.x. ↗
  29. Ana-Maria Popescu, Bao Nguyen, and Oren Etzioni. 2005. Opine: Extracting product features and opinions from reviews. In Proceedings of HLT/EMNLP interactive demonstrations. Association for Computational Linguistics, pages 32–33. https://doi.org/10.3115/1225733.1225750. ↗
  30. Chris Quirk, Chris Brockett, and William B Dolan. 2004. Proceedings of the 2004 conference on empirical methods in natural language processing. pages 142–149. http://aclweb.org/anthology/W04-3219. Language, Information, and Computation (PACLIC 27). https://aclweb.org/anthology/Y/Y13/Y13-1035.pdf. ↗ our card
  31. Janyce Wiebe, Theresa Wilson, Rebecca Bruce, Matthew Bell, and Melanie Martin. 2004. Learning subjective language. Computational linguistics 30(3):277–308. http://aclweb.org/anthology/J04-3002. ↗
  32. Sander Wubben, Antal Van Den Bosch, and Emiel Krahmer. 2010. Paraphrase generation as monolingual translation: Data and evaluation. In Proceedings of the 6th International Natural Language Generation Conference. Association for Computational Linguistics, pages 203–207. http://dblp.uni-trier.de/rec/bib/conf/inlg/WubbenBK10. ↗
  33. Wei Xu, Chris Callison-Burch, and William B Dolan. 2015. Semeval-2015 task 1: Paraphrase and semantic similarity in twitter (pit). Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015) https://doi.org/10.18653/v1/s15-2001. ↗ our card
  34. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 http://dblp2.uni-trier.de/rec/bib/journals/corr/abs-1212-5701. ↗ our card
  35. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. pages 704–714. http://aclweb.org/anthology/D13-1066. ↗ our card
  36. SG Shamay-Tsoory, Rachel Tomer, and Judith Aharon-Peretz. 2005. The neuroanatomical basis of understanding sarcasm and its relationship to social cognition. Neuropsychology 19(3):288. https://doi.org/10.1037/0894-4105.19.3.288. ↗
  37. FJ Stingfellow. 1994. The Meaning of Irony. New York: State University of NY. find
  38. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112. http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf. ↗
  39. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm-a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In ICWSM. http://dblp.uni-trier.de/rec/bib/conf/icwsm/TsurDR10. ↗
  40. Po-Ya Angela Wang. 2013. # irony or# sarcasma quantitative and qualitative study based on twitter. In Proceedings of the 27th Pacific Asia Conference on find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/peled-2017-sarcasm-sign-interpreting-sarcasm-sentiment/