Elamur Research⌕
☰
home / papers / 2022 / niksss at SemEval-2022 Task 6: Are Traditionally Pre-Trained Contextual Embeddings Enough for Detecting Intended Sarcasm ?

niksss at SemEval-2022 Task 6: Are Traditionally Pre-Trained Contextual Embeddings Enough for Detecting Intended Sarcasm ?

Paper · 2022 · Nikhil Singh

Summary

This paper describes one team's entry to SemEval-2022 Task 6, which targets intended sarcasm in English and Arabic. The labels come from the authors of the texts themselves, so they are less noisy than tag-based or third-party annotations. The system tests whether off-the-shelf contextual encoders can handle sarcasm without any sarcasm-specific design.

For the binary sarcastic vs. non-sarcastic subtask (A), each language gets a pretrained transformer: cardiffnlp/twitter-roberta-base-sentiment for English and AraBERT for Arabic. A dropout layer and a simple classifier sit on top. For the English-only irony-category subtask (B), the same encoder feeds a multilabel head trained with a label-ranking loss. Social-media noise was cleaned before training.

The system placed 10th (English) and 11th (Arabic) in Subtask A, with a sarcastic-class F1 of about 0.40 in both languages. Subtask B results were poor. The author argues that standard pretraining objectives do not encode how sarcasm works. He proposes future work that teaches a seq2seq model to rewrite sarcastic text into a literal paraphrase.

Key points

  • Subtask A ranking used the F1 of the sarcastic class: 40.16% for English (10th place) and 40% for Arabic (11th place).
  • Other Subtask A metrics were:
    • English: overall F1 0.6353, accuracy 0.7850.
    • Arabic: overall F1 0.58, accuracy 0.6571, recall 0.7167.
  • The encoders were cardiffnlp/twitter-roberta-base-sentiment and AraBERT, both producing 768-dimensional features, trained with AdamW at learning rate 2e-5 on a P100 in Google Colab Pro.
  • Subtask B was framed as multilabel classification with an LRAP loss (built with Simple Transformers). It ranked 22nd with a macro F1 of 0.0380.
  • The data had roughly a 3:1 skew toward non-sarcastic texts. Down-weighting that class still left the model biased toward it, while ensembling models trained with different hyperparameters helped after the evaluation period.

Interesting ideas

  • I1. The labels are supplied by the author of each text, so they capture what the writer intended rather than how a reader judged it, which reduces label noise for a phenomenon that is highly subjective.
  • I2. Starting from an encoder already tuned for sentiment is a cheap way to inject affect signals, since sarcasm often rests on a clash between surface sentiment and meaning.
  • I3. Pairing each sarcastic text with a literal paraphrase that means the same turns irony into a rewriting problem rather than a plain yes/no label.
  • I4. When irony subcategories are very close, the classifier's output probabilities cluster together, which signals that the label set is too fine for the available evidence.
  • I5. Ensembling models trained with different hyperparameters gave clear gains, suggesting that single-run results understate what the setup can do.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Irony test sets for evaluation could favour author-declared intent labels, so prag's irony detector is scored against intent rather than against annotator guesses.
  • I2 (frames): prag could compute an explicit sentiment polarity per clause and flag irony when it conflicts with the situation described in the frames, keeping the decision explainable.
  • I3 (back): The text → graph → text path could be tested on whether a sarcastic sentence and its literal paraphrase map to the same meaning graph, with generation of the literal version used as a check.
  • I4 (frames): Speech-act and irony subtypes in prag could be made hierarchical, so the system commits to "ironic" first and refines the subtype only when distinct cues are present.
  • I5 (learn): Witness-agreement for silver data could deliberately include parser variants differing only in hyperparameters, and their disagreement could be used as an uncertainty signal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Manipal University JaipurJaipur · India3 works · 235 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 3Research subjects and languages the work is about.English · 1,373 worksSentiment & emotion · 608 worksSemitic languages · 1,484 works

Cite

Cite as Nikhil Singh. 2022. niksss at SemEval-2022 Task 6: Are Traditionally Pre-Trained Contextual Embeddings Enough for Detecting Intended Sarcasm ?. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.127

@inproceedings{singh2022niksss,
  title = {{niksss at SemEval-2022 Task 6: Are Traditionally Pre-Trained Contextual Embeddings Enough for Detecting Intended Sarcasm ?}},
  author = {Nikhil Singh},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.127},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.127}
}

References

18 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSar-casmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. find
  2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  3. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644–1650, Online. Association for Computational Linguistics. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  4. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  5. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
  6. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  7. Tanvi Dadu and Kartikey Pant. 2020. Sarcasm detection using context separators in online discourse. In Proceedings of the Second Workshop on Figurative Language Processing, pages 51–55. ↗
  8. Rolandos Alexandros Potamias, Georgios Siolas, and Andreas-Georgios Stafylopatis. 2020. A transformer-based approach to irony and sarcasm detection. Neural Computing and Applications, 32(23):17309– 17320. ↗
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  10. Tomás Ptácek, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In COL-ING.
  11. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 392–398, Istanbul, Turkey. European Language Resources Association (ELRA). ↗
  12. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581–586, Portland, Oregon, USA. Association for Computational Linguistics.
  13. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016. Are word embedding-based features useful for sarcasm detection? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1006–1011, Austin, Texas. Association for Computational Linguistics. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ↗
  14. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. ↗
  15. Antonio Reyes, Paolo Rosso, and Davide Buscaldi. 2012. From humor recognition to irony detection: The figurative language of social media. Data Knowledge Engineering, 74:1–12. Applications of Natural Language to Information Systems. ↗
  16. Hamed Yaghoobian, Hamid R Arabnia, and Khaled Rasheed. 2021. Sarcasm detection: A comparative study. arXiv preprint arXiv:2107.02276. ↗
  17. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32. ↗
  18. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480–1489, San Diego, California. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singh-2022-niksss-semeval-2022-task-6/