Elamur Research⌕
☰
home / papers / 2022 / R2D2 at SemEval-2022 Task 6: Are language models sarcastic enough? Finetuning pre-trained language models to identify sarcasm

R2D2 at SemEval-2022 Task 6: Are language models sarcastic enough? Finetuning pre-trained language models to identify sarcasm

Paper · 2022 · Mayukh Sharma, Ilanthenral Kandasamy and Vasantha W B

Summary

The authors tackle sarcasm detection in the English part of iSarcasmEval (SemEval-2022 Task 6). Its data labels sarcasm by what the tweet's author intended, not by what annotators perceived. They entered three subtasks. Subtask A is binary sarcastic/non-sarcastic classification. Subtask B is multilabel tagging of irony type, with six possible labels: satire, sarcasm, irony, overstatement, understatement and rhetorical. In the third subtask, C, the system must pick out which of two meaning-equivalent texts is the sarcastic one. For A and B they fine-tuned several pre-trained transformers (BERT, RoBERTa, MPNet, ELECTRA) with small classification heads and class weighting against imbalance. For C they built a Siamese network with shared weights. It was trained jointly with cross-entropy and a loss that pushes the sarcastic and non-sarcastic representations apart. The team placed 7th in Subtask B and 8th in Subtask C, with a moderate result in Subtask A. All encoders performed about the same. Absolute scores on A and B stayed low, but pairwise discrimination in C reached roughly 75% accuracy. The study matters because it shows a gap: deciding in isolation whether text is sarcastic is far harder for these models than contrasting it with a literal paraphrase.

Key points

  • Data and imbalance.
    • Subtask A (English) has 3121 training texts, of which only 794 are sarcastic. Its test set has 1400 texts with 200 sarcastic.
    • Subtask B is severely skewed. For example, understatement has just 9 training examples.
    • Subtask C provides 780 training pairs of a sarcastic text and its rephrase.
  • Class weights. These were computed as N / (number of classes × class count) to counter the imbalance.
  • Architecture.
    • Each head is a 64-unit dense layer, then batch normalisation, then a sigmoid output.
    • Subtask B uses one such branch per irony category on top of shared encoder features.
    • Input is the [CLS] or <s> token representation, with sequences fixed at 70 tokens.
  • Official results.
    • Subtask B: MPNet reached macro-F1 0.076, ranked 7th and best on satire.
    • Subtask C: ELECTRA reached accuracy 0.750, ranked 8th.
    • Subtask A: MPNet reached sarcastic-class F1 of about 0.33.
  • Training setup. Models were built in Keras with Hugging Face transformers and trained for ten epochs on Colab TPUs. The checkpoint was chosen on the development set. Code is public on GitHub.

Interesting ideas

  • I1. Identifying sarcasm by contrasting a text with its literal paraphrase proved much easier than judging it alone, because the pair isolates what the figurative wording adds.
  • I2. A Siamese encoder with tied weights maps both texts into one shared feature space, and an extra objective pushes the two representations apart so their difference stands out.
  • I3. Each training pair was fed in both orders, (sarcastic, rephrase) and (rephrase, sarcastic), so neither head could cheat by learning a fixed position-to-label mapping.
  • I4. Rare categories were handled with one detection branch per label plus inverse-frequency class weights, rather than a single softmax over all labels.
  • I5. Very different encoders ended up with nearly identical scores, which suggests the limits came from the task and data rather than from the choice of architecture.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The irony module in prag could compare an utterance's frame against the frame of its literal reading and treat the mismatch as explainable evidence of irony.
  • I2 (learn): The same shared-encoder, maximise-the-contrast setup could train a scorer that tells a correct parse or frame apart from a minimally altered wrong one.
  • I3 (learn): When building silver pairs or negative controls, Mova should randomise or duplicate the order of inputs so evaluators cannot exploit position.
  • I4 (frames): Speech-act or irony-type labels with very few examples (such as understatement) could each get an independent weighted detector, which can be inspected and tuned separately.
  • I5 (learn): Mova's evaluations should report several model variants side by side, so that gains are not credited to one architecture when the data is the real bottleneck.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Vellore Institute of Technology UniversityVellore · India2 works · 237 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 English · 1,373 works Sentiment & emotion · 608 works

Cite

Cite as Mayukh Sharma, Ilanthenral Kandasamy and Vasantha W B. 2022. R2D2 at SemEval-2022 Task 6: Are language models sarcastic enough? Finetuning pre-trained language models to identify sarcasm. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.143

@inproceedings{sharma2022r2d2,
  title = {{R2D2 at SemEval-2022 Task 6: Are language models sarcastic enough? Finetuning pre-trained language models to identify sarcasm}},
  author = {Mayukh Sharma, Ilanthenral Kandasamy and Vasantha W B},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.143},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.143}
}

References

29 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. our card
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2016. Neural machine translation by jointly learning to align and translate. our card
  3. Christos Baziotis, Nikos Pelekis, and Christos Doulk-eridis. 2017. DataStories at SemEval-2017 task 4: Deep LSTM with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 747–754, Vancouver, Canada. Association for Computational Linguistics. our card
  4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗ our card
  5. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an _Obviously_ perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4619–4629, Florence, Italy. Association for Computational Linguistics. ↗ our card
  6. François Chollet et al. 2015. keras.io. Keras. https:// ↗
  7. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗ our card
  9. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rah-wan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1615–1625, Copenhagen, Denmark. Association for Computational Linguistics. our card
  10. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. CASCADE: Contextual sarcasm detection in online discussion forums. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1837–1848, Santa Fe, New Mexico, USA. Association for Computational Linguistics. our card
  11. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762, Beijing, China. Association for Computational Linguistics. ↗ our card
  12. Anupam Khattri, Aditya Joshi, Pushpak Bhattacharyya, and Mark Carman. 2015. Your sentiment precedes you: Using an author’s historical tweets to predict sarcasm. In Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 25–30, Lisboa, Portugal. Association for Computational Linguistics. ↗ our card
  13. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. ↗
  14. Gregory R. Koch. 2015. Siamese neural networks for one-shot image recognition. find
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
  16. Silviu Oprea and Walid Magdy. 2020. iSarcasm: A dataset of intended sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279–1289, Online. Association for Computational Linguistics. ↗ our card
  17. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗ our card
  18. XiPeng Qiu, TianXiang Sun, YiGe Xu, YunFan Shao, Ning Dai, and XuanJing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872– 1897. ↗ our card
  19. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714, Seattle, Washington, USA. Association for Computational Linguistics. ↗ our card
  20. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502– 518, Vancouver, Canada. Association for Computational Linguistics. our card
  21. Ranjan Satapathy, Claudia Guerreiro, Iti Chaturvedi, and Erik Cambria. 2017. Phonetic-based microtext normalization for twitter sentiment analysis. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pages 407–413. find
  22. Mayukh Sharma, Ilanthenral Kandasamy, and W.b. Vasantha. 2020. Memebusters at SemEval-2020 task 8: Feature fusion model for sentiment analysis on memes using transfer learning. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1163–1171, Barcelona (online). International Committee for Computational Linguistics. ↗
  23. Mayukh Sharma, Ilanthenral Kandasamy, and W.B. Vasantha. 2021a. Comparison of neutrosophic approach to various deep learning models for sentiment analysis. Knowledge-Based Systems, 223:107058. ↗
  24. Mayukh Sharma, Ilanthenral Kandasamy, and W.b. Vasantha. 2021b. YoungSheldon at SemEval-2021 task 7: Fine-tuning is all you need. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1146–1152, Online. Association for Computational Linguistics. ↗
  25. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. In NeurIPS 2020. ACM. ↗ our card
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. our card
  27. Byron C. Wallace, Do Kook Choe, and Eugene Charniak. 2015. Sparse, contextually informed models for irony detection: Exploiting user communities, entities and sentiment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1035–1044, Beijing, China. Association for Computational Linguistics. ↗ our card
  28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗ our card
  29. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. ↗ our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sharma-2022-r2d2-semeval-2022-task-6/