Elamur Research⌕
☰
home / papers / 2022 / stce at SemEval-2022 Task 6: Sarcasm Detection in English Tweets

stce at SemEval-2022 Task 6: Sarcasm Detection in English Tweets

Paper · 2022 · Mengfei Yuan, Zhou Mengyuan, Lianxin Jiang et al.

Summary

The paper describes the winning system for the English binary track (SubTask-A) of SemEval-2022 Task 6 (iSarcasmEval), which asks whether a tweet is sarcastic. The official English training set has 3468 tweets. Of these, 862 are sarcastic and come with author-written non-sarcastic rephrasings. The team enlarged this data with three public sarcasm and irony collections. Tweets were lightly cleaned: user mentions and links became placeholder tokens and slang abbreviations were expanded, but odd capitalization, elongated spellings and repeated punctuation were kept as signals. Hand-crafted cues were appended to each tweet as extra text segments. The classifiers are large pretrained transformers fine-tuned with a multi-sample dropout head, adversarial embedding perturbation and an auxiliary contrastive loss. The final output is a vote over 15 such models that mixes hard and soft voting. The submission reached a sarcastic-class F1 of 0.6052 and placed first of 43 teams. Ablations attribute gains to external data, multi-sample dropout, the textual features and tuning of the contrastive loss.

Key points

  • Data. The official data was extended with:
    • iSarcasm: 563 sarcastic and 2279 non-sarcastic tweets.
    • A random 10,000-text sample of the Multi-modal Sarcasm data (33,859 items, text only). The authors sampled to save training time and to avoid bias from a very large added corpus.
    • The SemEval-2018 Task 3 tweets (3800).
  • Encoders. RoBERTa-large, DeBERTa-v3-large and XLM-RoBERTa-large were used. XLM-RoBERTa-large was used when the Arabic tweets were mixed into training.
  • Classifier head and training.
    • Head: four dropout branches at rate 0.4 (0.2 for DeBERTa) after pooling.
    • Optimizer: AdamW with a learning rate of 1e-05.
    • Adversarial training: FGM, chosen over PGD for similar scores at lower cost.
    • Loss: cross-entropy plus NT-Xent (temperature 0.2), with the contrastive part weighted at 0.1.
  • Appended features. These were joined to the tweet with </s></s> separators:
    • emoji;
    • adjectives and adverbs mined from sarcastic/rephrase differences, such as "really", "never" and "actually";
    • elongated or mid-sentence capitalized words;
    • transitional words;
    • AFINN polarity.
  • Results. The best single configuration (XLM-RoBERTa-large on all data with features) scored 0.6029 sarcastic F1 and 0.7676 Macro F1. The final ensemble scored 0.6052 sarcastic F1 and 0.7675 Macro F1.

Interesting ideas

  • I1. Using paired sarcastic tweets and their literal rephrasings to extract which words disappear in the literal version yields a small cue lexicon for sarcasm almost for free.
  • I2. Normalize abbreviations but deliberately preserve elongations, odd capitals and repeated punctuation, because these carry emphasis rather than noise.
  • I3. Mixed voting uses a plain majority when models clearly agree (vote margin above 2) and falls back to averaged probabilities when the vote is close.
  • I4. Subsampling a large external corpus instead of adding all of it limits the risk that out-of-domain data dominates the target task.
  • I5. Feeding hand-crafted cues to a neural model as appended text segments, rather than as a separate vector, keeps the features human-readable in the input.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Irony detection in prag could mine minimal literal/ironic pairs to learn explicit, inspectable cue words (intensifiers, "actually", "never") that trigger an irony hypothesis.
  • I2 (tokens): The tokenizer could map "soooo" to the lemma "so" while recording an emphasis attribute, so later stages still see the exaggeration signal.
  • I3 (learn): Silver-data agreement among witness parsers could accept clear majorities directly and use confidence-weighted arbitration only on contested arcs.
  • I4 (learn): When mixing treebanks or LLM-teacher silver data, Mova could cap each source's share so the target dialect is not swamped.
  • I5 (answer): Readers could attach explicit evidence strings (detected cues, polarity) to their inputs, so the answer stays traceable to named signals.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 English · 1,373 works Sentiment & emotion · 608 works

Cite

Cite as Mengfei Yuan, Zhou Mengyuan, Lianxin Jiang, Yang Mo and Xiaofeng Shi. 2022. stce at SemEval-2022 Task 6: Sarcasm Detection in English Tweets. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.113

@inproceedings{yuan2022stce,
  title = {{stce at SemEval-2022 Task 6: Sarcasm Detection in English Tweets}},
  author = {Mengfei Yuan, Zhou Mengyuan, Lianxin Jiang, Yang Mo and Xiaofeng Shi},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.113},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.113}
}

References

36 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. our card
  2. Santosh Kumar Bharti, Bakhtyar Vachha, RK Pradhan, Korra Sathya Babu, and Sanjay Kumar Jena. 2016. Sarcastic sentiment detection in tweets streamed in real time: a big data approach. Digital Communications and Networks, 2(3):108–121. ↗
  3. Konstantin Buschmeier, Philipp Cimiano, and Roman Klinger. 2014. An impact analysis of features in a classification approach to irony detection in product reviews. In Proceedings of the 5th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 42–49. our card
  4. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR. ↗ our card
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. ↗
  6. Irazú Hernández-Farías, José-Miguel Benedí, and Paolo Rosso. 2015. Applying basic features from sentiment analysis for automatic irony detection. In Iberian Conference on Pattern Recognition and Image Analysis, pages 337–344. Springer. find
  7. Hiroshi Inoue. 2019. Multi-sample dropout for accelerated training and better generalization. arXiv preprint arXiv:1905.09788. ↗
  8. Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2019. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. arXiv preprint arXiv:1911.03437. ↗ our card
  9. Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661–18673. ↗
  10. Avinash Kumar, Vishnu Teja Narapareddy, Veerub-hotla Aditya Srikanth, Aruna Malapati, and Lalita Bhanu Murthy Neti. 2020. Sarcasm detection using multi-head attention based bidirectional lstm. Ieee Access, 8:6388–6397. ↗
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  12. Ruth Filik, Alexandra T , urcan, Dominic Thompson, Nicole Harvey, Harriet Davies, and Amelia Turner. 2016. Sarcasm and emoticons: Comprehension and emotional impact. Quarterly Journal of Experimental Psychology, 69(11):2130–2146. find
  13. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. ↗ our card
  14. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. ↗ our card
  15. Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2016. Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725. ↗ our card
  16. Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE. find
  17. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020a. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738. our card
  18. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543. ↗ our card
  19. Silviu Oprea and Walid Magdy. 2020. isarcasm: A dataset of intended sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. our card
  20. Reynier Ortega-Bueno, Francisco Rangel, D Hernández Farıas, Paolo Rosso, Manuel Montes-y Gómez, and José E Medina Pagola. 2019. Overview of the task on irony detection in spanish variants. In Proceedings of the Iberian languages evaluation forum (IberLEF 2019), co-located with 34th conference of the Spanish Society for natural language processing (SEPLN 2019). CEUR-WS. org, volume 2421, pages 229–256. find
  21. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. arXiv preprint arXiv:1610.08815. ↗ our card
  22. Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. 2019. Adversarial robustness through local linearization. Advances in Neural Information Processing Systems, 32. ↗
  23. Antonio Reyes and Paolo Rosso. 2014. On the difficulty of automatically detecting irony: beyond a simple case of negation. Knowledge and Information Systems, 40(3):595–614. ↗
  24. Dymitr Ruta and Bogdan Gabrys. 2005. Classifier selection for majority voting. Information fusion, 6(1):63– 81. ↗
  25. Hamed Yaghoobian, Hamid R Arabnia, and Khaled Rasheed. 2021. Sarcasm detection: A comparative study. arXiv preprint arXiv:2107.02276. ↗
  26. Shiwei Zhang, Xiuzhen Zhang, Jeffrey Chan, and Paolo Rosso. 2019. Irony detection via sentiment-based transfer learning. Information Processing & Management, 56(5):1633–1644. ↗
  27. Yong Zhang, Hongrui Zhang, Jing Cai, and Binbin Yang. 2014. A weighted voting classifier based on differential evolution. In Abstract and Applied Analysis, volume 2014. Hindawi. ↗
  28. Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2019. Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint arXiv:1909.11764. ↗ our card
  29. Samer Muthana Sarsam, Hosam Al-Samarraie, Ahmed Ibrahim Alzahrani, and Bianca Wright. 2020. Sarcasm detection using machine learning algorithms in twitter: A systematic review. International Journal of Market Research, 62(5):578–598. ↗
  30. Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghi-asi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. 2019. Adversarial training for free! Advances in Neural Information Processing Systems, 32. ↗
  31. Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29. find
  32. Devendra Kr Tayal, Sumit Yadav, Komal Gupta, Bhawna Rajput, and Kiran Kumari. 2014. Polarity detection of sarcastic political tweets. In 2014 International conference on computing for sustainable global development (INDIACom), pages 625–628. IEEE. find
  33. Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv e-prints, pages arXiv–1807. our card
  34. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50. our card
  35. Palak Verma, Neha Shukla, and AP Shukla. 2021. Techniques of sarcasm detection: A review. In 2021 International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), pages 968–972. IEEE. find
  36. Feng Wang and Huaping Liu. 2021. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2495–2504. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuan-2022-stce-semeval-2022-task-6/