Elamur Research⌕
☰
home / papers / 2022 / TUG-CIC at SemEval-2021 Task 6: Two-stage Fine-tuning for Intended Sarcasm Detection

TUG-CIC at SemEval-2021 Task 6: Two-stage Fine-tuning for Intended Sarcasm Detection

Paper · 2022 · Jason Angel, Segun Taofeek Aroyehun and Alexander Gelbukh

Summary

The paper describes the TUG-CIC entry for iSarcasmEval (SemEval-2022 Task 6), Subtask A in English. The task is to decide whether a tweet was meant as sarcastic by its own author. The organizers' data has about one sarcastic tweet in four, and its labels come from the authors themselves rather than from hashtags or outside annotators.

The team built a staged transfer pipeline on BERTweet variants:

  • an optional starting checkpoint already tuned for sentiment or emotion,
  • then an intermediate fine-tuning pass on the intended-sarcasm portion of SPIRS,
  • and finally training on the shared-task data.

Their best submission, BERTweet-large passed through SPIRS, reached 0.530 F1 on the sarcastic class and placed third of 43 teams. Later ablations undercut the motivating hypothesis. Sentiment and emotion checkpoints scored below plain BERTweet, and skipping the SPIRS stage did not hurt.

A small manual error study names three hard cases: missing conversational context, required world knowledge, and polarity clashes between words and emoji. The work matters as an honest benchmark of how far tweet-tuned transformers get on author-intended irony.

Key points

  • Shared-task data: 3,468 training tweets (867 sarcastic) and 345 validation tweets (86 sarcastic). The ranking metric is F1 on the positive (sarcastic) class only.
  • SPIRS subset: only intended-sarcastic and negative examples were kept, giving 15,950 train and 1,773 dev tweets. Perceived-sarcasm tweets were dropped as a different task.
  • Backbones: BERTweet-base, BERTweet-large (vinai) and the cardiffnlp bertweet-base-sentiment / -emotion checkpoints.
  • Training setup:
    • adapter-transformers library, with a two-layer dense head on the pooled output;
    • AdamW, batch size 64, learning rate 1e-5, warmup ratio 0.1, up to 15 epochs;
    • early stopping on validation F1, checked every 20 steps, patience 5;
    • regularization by weight decay 0.01, dropout 0.2 and label smoothing 0.1.
  • Test results:
    • BERTweet-large-spirs scored 0.530 F1. In post-evaluation, BERTweet-large without SPIRS scored 0.535, and base without SPIRS scored 0.500 against 0.468 with it.
    • Base tuned from the sentiment or emotion checkpoints fell to 0.459 / 0.473.
    • Test recall (~0.7) was far above precision (~0.3–0.4), so the systems over-flag sarcasm.

Interesting ideas

  • I1. Separating the speaker's intent from a reader's perception, and training only on the intent side, treats the two as distinct targets instead of mixing their labels.
  • I2. The post-hoc ablations showed that neither the intermediate pretraining nor the sentiment/emotion initialization explained the gain, which a submission-only comparison would have hidden.
  • I3. One error class is a lexical cue clashing with an emoji (e.g. "thanks" followed by a sad face), meaning polarity disagreement between channels signals irony.
  • I4. Another error class needs knowledge about the world (drivers ignoring turn signals, three followers being unimpressive), which surface models cannot supply.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): silver irony or speech-act labels could carry a tag for whose viewpoint they encode (author, reader, teacher LLM), so conflicting perspectives are not merged into one target.
  • I2 (learn): every added training stage should be checked against a run without it, as a negative control, before it is credited with an improvement.
  • I3 (frames): the irony module in prag could compute polarity for words and emoji separately and raise an explainable irony hypothesis when they disagree.
  • I4 (world): comparing a stated evaluation with expected states from the knowledge graph (e.g. a low follower count is not a big event) could let irony detection cite the violated expectation as its explanation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Instituto Politécnico NacionalMexico City · Mexico18 works · 4,307 citations🎓Graz University of TechnologyGraz · Austria3 works · 5 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,379 worksSentiment & emotion · 1,035 works

Cite

Cite as Jason Angel, Segun Taofeek Aroyehun and Alexander Gelbukh. 2022. TUG-CIC at SemEval-2021 Task 6: Two-stage Fine-tuning for Intended Sarcasm Detection. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.133

@inproceedings{angel2022tug,
  title = {{TUG-CIC at SemEval-2021 Task 6: Two-stage Fine-tuning for Intended Sarcasm Detection}},
  author = {Jason Angel, Segun Taofeek Aroyehun and Alexander Gelbukh},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.133},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.133}
}

References

16 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics.
  2. Mondher Bouazizi and Tomoaki Ohtsuki. 2015. Opinion mining in twitter how to make use of sarcasm to enhance sentiment analysis. In Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2015, pages 1594–1597. find
  3. Diana G Maynard and Mark A Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Lrec 2014 proceedings. ELRA.
  4. Pulkit Mehndiratta and Devpriya Soni. 2019. Identification of sarcasm using word embeddings and hyperparameters tuning. Journal of Discrete Mathematical Sciences and Cryptography, 22(4):465–489. ↗
  5. Silviu Vlad Oprea and Walid Magdy. 2020. The effect of sociocultural variables on sarcasm communication online. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW1):1–22. ↗
  6. Boaz Shmueli, Lun-Wei Ku, and Soumya Ray. 2020. Reactive Supervision: A New Method for Collecting Sarcasm Data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2553–2559, Online. Association for Computational Linguistics. ↗
  7. Himani Srivastava, Vaibhav Varshney, Surabhi Kumari, and Saurabh Srivastava. 2020. A novel hierarchical bert architecture for sarcasm detection. In Proceedings of the Second Workshop on Figurative Language Processing, pages 93–97.
  8. Haiyang Wang, Xin Song, Bin Zhou, Ye Wang, Liqun Gao, and Yan Jia. 2021. Performance evaluation of pre-trained models in sarcasm detection task. In International Conference on Web Information Systems Engineering, pages 67–75. Springer. find
  9. Simona Frenda, Alessandra Teresa Cignarella, Valerio Basile, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2022. The unbearable hurtfulness of sarcasm. Expert Systems with Applications, page 116398. ↗
  10. Setra Genyang Wicana, Taha Yasin İbisoglu, and Uraz Yavanoglu. 2017. A review on sarcasm detection from machine-learning perspective. In 2017 IEEE 11th International Conference on Semantic Computing (ICSC), pages 469–476. IEEE. find
  11. Debanjan Ghosh, Ritvik Shrivastava, and Smaranda Muresan. 2021. " laughing at you or with you": The role of sarcasm in shaping the disagreement space. arXiv preprint arXiv:2101.10952. ↗
  12. Uraz Yavanoglu, Taha Yasin Ibisoglu, and Setra Genyang Wıcana. 2018. Sarcasm detection algorithms. International Journal of Semantic Computing, 12(03):457–478. ↗
  13. Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), 50(5):1–22. ↗
  14. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762.
  15. Sandeepa Kannangara. 2018. Mining twitter for fine-grained political opinion polarity classification, ideology detection and sarcasm detection. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 751–752. find
  16. Bing Liu et al. 2010. Sentiment analysis and subjectivity. Handbook of natural language processing, 2(2010):627–666. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/angel-2022-tug-cic-semeval-2021-task/