Elamur Research⌕
☰
home / papers / 2020 / Sarcasm Detection Using an Ensemble Approach

Sarcasm Detection Using an Ensemble Approach

Paper · 2020 · Jens Lemmens, Ben Burtenshaw, Ehsan Lotfi et al.

Summary

This paper addresses sarcasm classification of social-media replies. The task was a shared task at the Second Workshop on Figurative Language Processing (ACL 2020), where each reply comes with the earlier turns of its conversation. The organizers supplied balanced, automatically labelled data: 5,000 tweets and 4,400 Reddit comments for training, plus 1,800 of each for testing.

The authors built four deliberately different classifiers:

  • a bidirectional LSTM over GloVe, Emoji2Vec and hashtag-prediction embeddings;
  • a CNN-LSTM enriched with punctuation, casing, Vader sentiment and stop-word counts;
  • an MLP on Infersent sentence vectors;
  • a liblinear SVM over POS, function-word, NRC emotion and character n-grams.

All except the SVM also read the two preceding turns. Out-of-fold predictions from 10-fold cross-validation feed an AdaBoost meta-classifier built on decision trees. The meta-classifier also sees text lengths, data source and sentiment, so it can learn which component to trust for which kind of input.

The ensemble reaches F1 of 66.7% on Reddit and 74.0% on Twitter, beating every single component. This shows that heterogeneous, context-aware detectors complement one another.

Key points

  • Data: 5,000 Twitter and 4,400 Reddit training responses, plus 1,800 test responses per platform. All sets are balanced and labelled automatically. User mentions and links are masked as @USER and <URL>.
  • Context window: two preceding turns, chosen because every instance was guaranteed at least that much. The SVM ignores context and models only the reply's style and emotion.
  • Meta-model: scikit-learn AdaBoost with a decision-tree base estimator. Its inputs are the component predictions plus:
    • character lengths of the response and context;
    • source platform;
    • Vader sentiment for the reply and both prior turns.
  • Results:
    • Ensemble cross-validation F1 on the training data is 76.9%.
    • On the official test sets, the best single model is the LSTM on Reddit (63.5) and the MLP on Twitter (68.3). The ensemble scores 66.7 and 74.0 respectively.
    • In an ablation, dropping any component lowered the scores.
  • Reddit was harder than Twitter for every model in both precision and recall. The authors tentatively blame the unrestricted length of Reddit comments.

Interesting ideas

  • I1. Instead of averaging votes, train an arbiter that also sees cheap properties of the input (length, source, polarity), so it learns which expert is reliable in which situation.
  • I2. Train the stacking layer only on out-of-fold predictions, so each component's output on a sentence comes from a model that never saw that sentence and the meta-learner is not fooled by overfit confidence.
  • I3. Score sentiment separately for the reply and for each preceding turn, letting the model notice when an upbeat reply sits on top of a negative situation.
  • I4. Use an encoder trained on natural language inference because sarcasm often hinges on a logical clash between what is said and what the context implies.
  • I5. Confirm through ablation that each ensemble member actually contributes, rather than assuming diversity helps.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): the agreement step over witness parsers could use such an arbiter, keyed on sentence length or genre, to decide whose tree to trust for silver data rather than requiring blind consensus.
  • I2 (learn): any learned combiner over parser or teacher outputs should be fitted on held-out predictions in the same way, to avoid inflated agreement.
  • I3 (frames): the irony module in prag could compare the polarity of the speaker's evaluative act with the polarity of the state described in prior discourse and flag mismatches as candidate sarcasm.
  • I4 (world): an explicit contradiction check between a reply's asserted state and the entity states already built for the text could serve as an explainable irony signal.
  • I5 (learn): when adding a new witness parser or rule source, run a leave-one-out evaluation to prove it improves results before keeping it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AntwerpAntwerp · Belgium9 works · 3,209 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 608 works

Cite

Cite as Jens Lemmens, Ben Burtenshaw, Ehsan Lotfi, Ilia Markov and Walter Daelemans. 2020. Sarcasm Detection Using an Ensemble Approach. Proceedings of the Second Workshop on Figurative Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.figlang-1.36

@inproceedings{lemmens2020sarcasm,
  title = {{Sarcasm Detection Using an Ensemble Approach}},
  author = {Jens Lemmens, Ben Burtenshaw, Ehsan Lotfi, Ilia Markov and Walter Daelemans},
  booktitle = {Proceedings of the Second Workshop on Figurative Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.figlang-1.36},
  url = {https://doi.org/10.18653/v1/2020.figlang-1.36}
}

References

23 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ilia Markov et al. 2020. Exploring stylometric and emotion-based features for multilingual cross-domain hate speech detection (in preparation). find
  2. Silvio Amir, Byron C. Wallace, Hao Lyu, and Paula Carvalho Mario J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 167–177, Berlin, Germany. Association for Computational Linguistics.
  3. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O’Reilly Media Inc. find
  4. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.
  5. Ben Eisner, Tim Rocktäschel, Isabelle Augenstein, Matko Bošnjak, and Sebastian Riedel. 2016. Emoji2vec: Learning emoji representations from their description. In Proceedings of The Fourth International Workshop on Natural Language Processing for Social Media, pages 48–54, Austin, Texas, USA. Association for Computational Linguistics.
  6. Aniruddha Ghosh and Tony Veale. 2016. Fracking sarcasm using neural network. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 161–169, San Diego, California. Association for Computational Linguistics. ↗
  7. Debanjan Ghosh, Alexander R. Fabbri, and Smaranda Muresan. 2018. Sarcasm analysis using conversation context. Computational Linguistics, 44(4):755– 792. ↗
  8. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581–586, Portland, Oregon, USA. Association for Computational Linguistics.
  9. Hongyan Jing, Daniel Lopresti, and Chilin Shih. 2003. Summarization of noisy documents: A pilot study. In Proceedings of the HLT-NAACL 03 Text Summarization Workshop, pages 25–32. find
  10. Aditya Joshi, Pushpak Bhattacharyya, Mark Carman, Jaya Saraswati, and Rajita Shukla. 2016. How do cultural differences impact the quality of sarcasm annotation?: A case study of Indian annotators and American text. In Proceedings of the 10th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (LaTeCH) 2016, pages 95–99, Berlin, Germany. Association for Computational Linguistics.
  11. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys, 50(5):73. ↗
  12. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodra-halli. 2018. A large self-annotated corpus for sarcasm. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). ↗
  13. Saif Mohammad and Peter Turney. 2013. Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29:436–465. ↗
  14. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(0):2825–2830. ↗
  15. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representations. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. find
  16. Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher Manning. 2018. Universal dependency parsing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 160–170, Brussels, Belgium. Association for Computational Linguistics.
  17. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714, Seattle, Washington, USA. Association for Computational Linguistics.
  18. Vered Shwartz, Gabriel Stanovsky, and Ido Dagan. 2017. Acquiring predicate paraphrases from news tweets. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 155–160, Vancouver, Canada. Association for Computational Linguistics. find
  19. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. We usually don’t like going to the dentist: Using common sense to detect irony on Twitter. Computational Linguistics, 44(4):793–832. ↗
  20. Byron C. Wallace, Do Kook Choe, Laura Kertz, and Eugene Charniak. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 512–516, Baltimore, Maryland, USA. Association for Computational Linguistics.
  21. Zelin Wang, Zhijian Wu, Ruimin Wang, and Yafeng Ren. 2015. Twitter sarcasm detection exploiting a context-based model. In Proceedings, Part I, of the 16th International Conference on Web Information Systems Engineering — WISE 2015 - Volume 9418, page 77–91, Berlin, Heidelberg. Springer-Verlag. find
  22. Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. 2016. Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3485–3495, Osaka, Japan. ↗
  23. The COLING 2016 Organizing Committee. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lemmens-2020-sarcasm-detection-ensemble-approach/