Elamur Research⌕
☰
home / papers / 2022 / X-PuDu at SemEval-2022 Task 6: Multilingual Learning for English and Arabic Sarcasm Detection

X-PuDu at SemEval-2022 Task 6: Multilingual Learning for English and Arabic Sarcasm Detection

Summary

The paper presents X-PuDu, a system for SemEval-2022 Task 6 (iSarcasmEval), which detects intended sarcasm in English and Arabic tweets. The task data were labelled by the tweet authors, who also wrote non-sarcastic rephrasings of their sarcastic posts. There are three subtasks:

  • Task A: binary detection on a single text.
  • Task B: English-only multi-label tagging into six ironic-speech categories.
  • Task C: picking the sarcastic member of a sarcastic/rephrase pair.

The team fine-tuned pre-trained Transformers (ERNIE-M for both languages, DeBERTa for English) with a classification head on the [CLS] token. In Task C, both texts were joined with [SEP] and fed to a cross-encoder. For Tasks A and C, they also tried multilingual learning, merging English and Arabic data into one fine-tuning run of ERNIE-M. Results came from ensembles averaged over 10-fold cross-validation and several hyperparameter settings.

The system placed 2nd/43 (Task A English), 5th/22 (Task B) and 1st/16 (Task C English). Joint bilingual training clearly helped on the paired task but not on single-sentence detection. This suggests that a contrastive pair exposes the ironic signal more readily.

Key points

  • The training set has 3468 texts, of which 867 are sarcastic and 2601 are not. Task B labels are very skewed: sarcasm 713, irony 155, rhetorical_question 101, overstatement 40, satire 25, understatement 10.
  • The official metrics differ by subtask: Task A uses F1 on the sarcastic class, Task B uses macro-F1, and Task C uses accuracy.
  • Submitted scores were:
    • Task A: DeBERTa reached 56.91 F1 in English and ERNIE-M reached 41.87 in Arabic.
    • Task B: DeBERTa scored 7.99, as reported.
    • Task C: DeBERTa reached 87.00 accuracy in English and monolingual ERNIE-M reached 84.00 in Arabic.
  • Effect of multilingual ERNIE-M:
    • Task C rose to 90.50 in Arabic (from 84.00) and to 82.50 in English (from 75.00). The Arabic score would have ranked about 2nd, but it was not submitted in time.
    • Task A stayed roughly level with the monolingual setting (40.36 vs 41.87 in Arabic).
  • Training setup:
    • Optimiser: AdamW with weight decay 0.01 and 10% warmup.
    • Learning rates: peak values of 1e-5 and 5e-6.
    • Regularisation: dropout 0.15.
    • Batching: effective batch 64 via gradient accumulation on 8 V100 GPUs.
    • Ensembling: averaged over 10 folds × learning rates × 20/30 epochs.
    • External data: adding Kaggle data with "figurative" and "irony" folded into "sarcasm" brought no gain.

Interesting ideas

  • I1. Deciding which of two near-paraphrases is ironic proved much easier than judging one sentence alone, because the contrast isolates the ironic cue. — For Mova (frames): the irony detector in prag could compare a sentence's frames with those of a literal paraphrase, flagging irony where the two frames diverge in polarity or intensity.
  • I2. The dataset pairs each sarcastic post with its author's literal restatement, giving a direct record of the intended meaning. — For Mova (frames): such pairs could serve as gold targets for implicature, testing whether the derived "what was really meant" matches the author's rephrase.
  • I3. Merging loosely related labels from an outside corpus added annotator noise rather than signal, even when training data were scarce. — For Mova (learn): before folding external or LLM-tier labels into silver data, check their category mapping against a held-out control instead of assuming more data helps.
  • I4. With little data, the authors used k-fold cross-validation for model choice and averaged all fold models for the final prediction, rather than trusting one random split. — For Mova (learn): evaluations on small school-test or fable QA sets could report fold-averaged scores to avoid mistaking split luck for real repair-rule gains.
  • I5. Training one model on the same objective in two languages helped where the task structure was strong (pairs) but not elsewhere. — For Mova (back): when translating through the meta-world, aligned bilingual pairs might be most useful for contrastive, structured checks rather than for single-sentence classifiers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

5 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 3 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.140 · arXiv 2211.16883

@inproceedings{han2022x,
  title = {{X-PuDu at SemEval-2022 Task 6: Multilingual Learning for English and Arabic Sarcasm Detection}},
  author = {Han, Yaqian and Chai, Yekun and Wang, Shuohuan and Sun, Yu and Huang, Hongyi and Chen, Guanghao and Xu, Yitong and Yang, Yang},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.140},
  eprint = {2211.16883},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.140}
}

References

23 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. find our card
  2. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mário J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. CoRR, abs/1607.00976. ↗ our card
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  4. Tanvi Dadu and Kartikey Pant. 2020. Sarcasm detection using context separators in online discourse. In Proceedings of the Second Workshop on Figurative Language Processing, pages 51–55. find our card
  5. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcasm in Twitter and Amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 107–116, Uppsala, Sweden. Association for Computational Linguistics. find our card
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  7. Debanjan Ghosh and Smaranda Muresan. 2018. "with 1 follower I must be AWESOME : P". exploring the role of irony markers in irony recognition. CoRR, abs/1804.05253. ↗
  8. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581–586, Portland, Oregon, USA. Association for Computational Linguistics. find our card
  9. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. find our card
  10. Soroush Javdan, Behrouz Minaei-Bidgoli, et al. 2020. Applying transformers and aspect-based sentiment analysis approaches on sarcasm detection. In Proceedings of the Second Workshop on Figurative Language Processing, pages 67–71. find our card
  11. Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie-m: enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora. arXiv preprint arXiv:2012.15674. ↗
  12. Rolandos Alexandros Potamias, Georgios Siolas, and Andreas-Georgios Stafylopatis. 2020. A transformer-based approach to irony and sarcasm detection. Neural Computing and Applications, 32(23):17309– 17320. find our card
  13. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929–1958. find
  14. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223. ↗ our card
  15. Yi Tay, Luu Anh Tuan, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. CoRR, abs/1805.02856. ↗ our card
  16. Joseph Tepperman, David Traum, and Shrikanth Narayanan. 2006. Yeah right: Sarcasm recognition for spoken dialogue systems. find
  17. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1804.07461. ↗ our card
  18. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 347–354, Vancouver, British Columbia, Canada. Association for Computational Linguistics. find our card
  19. Aditya Joshi, Pushpak Bhattacharyya, and Mark James Carman. 2016. Automatic sarcasm detection: A survey. CoRR, abs/1602.03426. ↗ our card
  20. Akshay Khatri, Pranav P, and Anand Kumar M. 2020. Sarcasm detection in tweets with BERT and glove embeddings. CoRR, abs/2006.11512. ↗
  21. John S. Leggitt and Raymond W. Gibbs. 2000. Emotional reactions to verbal irony. Discourse Processes, 29(1):1–24. ↗
  22. Ilya Loshchilov and Frank Hutter. 2017. weight decay regularization in adam. abs/1711.05101. ↗
  23. Fixing CoRR, Silviu Oprea and Walid Magdy. 2020. isarcasm: A dataset of intended sarcasm. ArXiv, abs/1911.03123. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/han-2022-x-pudu-semeval-2022-task/