Elamur Research⌕
☰
home / papers / 2022 / CS-UM6P at SemEval-2022 Task 6: Transformer-based Models for Intended Sarcasm Detection in English and Arabic

CS-UM6P at SemEval-2022 Task 6: Transformer-based Models for Intended Sarcasm Detection in English and Arabic

Summary

The paper describes the CS-UM6P entry to iSarcasmEval (SemEval-2022 Task 6). The task targets intended sarcasm: tweet authors label their own posts and also supply a literal rewording of any sarcastic one. The team built three classifiers on top of tweet-adapted encoders, Twitter-XLM-Roberta-base for English and MARBERT for Arabic:

  • a binary head;
  • a two-class softmax head;
  • a GAN-BERT-style model with a class-conditional generator.

Each model pairs an attention layer over token embeddings with the pooled output. The team compared plain, class-weighted and focal losses, and trained with and without the authors' rephrases. Sub-task C, which asks which of two texts is the sarcastic one, was solved by comparing the sarcasm probabilities that the sub-task A models assign to each text.

Adding rephrases as extra non-sarcastic examples helped, most strongly for Arabic. The simple heads beat the GAN variant. Ensembling models trained on different data variants lowered the official scores. Even so, the system placed first on Arabic sarcasm detection and second on the fine-grained category task. This shows that tweet-specific encoders and paraphrase-based negatives are effective for this subjective, low-resource problem.

Key points

  • Data: the training sets are 3,468 English tweets (867 sarcastic) and 3,102 Arabic tweets (745 sarcastic), labeled by their authors. Test sets have 1,400 items per language for sub-task A and 200 pairs for sub-task C.
  • Official ranks: 1st on A-Arabic, 15th on A-English, 2nd on B (English multi-label), 7th on C-Arabic and 11th on C-English.
  • Best non-official sarcastic-class F1:
    • 0.6217 on Arabic (Model 2 with focal loss, trained with rephrases);
    • 0.3833 on English (Model 1 with BCE).
  • Sub-task C accuracy: the best single models reached 0.85 (AR) and 0.71 (EN), versus 0.78 and 0.69 for the submitted ensemble. Mixing models trained with and without rephrases caused this drop.
  • Training setup:
    • focal loss with γ=2, α=0.8;
    • lr 1e-5, 10 epochs, batch 16;
    • Arabic input formatted as [SEP] dialect [SEP] text [SEP].
  • Sub-task B weakness: understatement and overstatement got an F1 of 0.

Interesting ideas

  • I1. A detector trained on single texts can solve a pairwise "which one is sarcastic" task by simply comparing its scores on the two texts, with no dedicated pair model needed. — For Mova (frames): the irony module in prag could score a sentence against its literal paraphrase and flag irony when the gap is large, giving an explainable contrastive signal.
  • I2. Literal rewordings written by the authors themselves make high-quality negatives, because they keep the topic and meaning but drop the figurative form. — For Mova (learn): pairing ironic sentences with literal paraphrases from an LLM teacher would give minimal-pair training and negative controls for the implicature and irony components.
  • I3. Metadata such as dialect can be passed to the encoder as an extra segment before the text, rather than training a separate model for each variety. — For Mova (tree): gparse could take a treebank, genre or UD-dialect tag as a prefix segment, letting one parser adapt its output per source.
  • I4. Hard-voting over models trained on different data distributions hurt results, so ensemble members should share a training regime or be weighted by validation scores. — For Mova (learn): when merging witness parsers into silver data, check that the witnesses were not trained on mismatched data before letting them vote equally.
  • I5. Reporting F1 per category exposed rare classes that scored zero, which a macro average alone would hide. — For Mova (learn): evaluation of speech-act or irony subtypes should report per-label scores so that silently failing rare categories are caught early.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

2 papers of our library cite it (of the 140884 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.117 · arXiv 2206.08415

@inproceedings{mahdaouy2022cs,
  title = {{CS-UM6P at SemEval-2022 Task 6: Transformer-based Models for Intended Sarcasm Detection in English and Arabic}},
  author = {Mahdaouy, Abdelkader El and Mekki, Abdellah El and Essefar, Kabil and Skiredj, Abderrahman and Berrada, Ismail},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.117},
  eprint = {2206.08415},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.117}
}

References

26 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗ our card
  2. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. find our card
  3. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 296–305, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find our card
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. find our card
  5. Francesco Barbieri, Luis Espinosa-Anke, and Jose Camacho-Collados. 2021. A Multilingual Language Model Toolkit for Twitter. In arXiv preprint arXiv:2104.12250. ↗
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  7. Danilo Croce, Giuseppe Castellucci, and Roberto Basili. 2020. GAN-BERT: Generative adversarial learning for robust text classification with a bunch of labeled examples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2114–2119, Online. Association for Computational Linguistics. ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  9. Abdelkader El Mahdaouy, Abdellah El Mekki, Kabil Essefar, Nabil El Mamoun, Ismail Berrada, and Ahmed Khoumsi. 2021. Deep multi-task model for sarcasm detection and sentiment analysis in Arabic language. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 334–339, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find our card
  10. Nabil El Mamoun, Abdelkader El Mahdaouy, Abdellah El Mekki, Kabil Essefar, and Ismail Berrada. 2021. CS-UM6P at SemEval-2021 task 1: A deep learning model-based pre-trained transformer encoder for lexical complexity. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 585–589, Online. Association for Computational Linguistics. find
  11. Abdellah El Mekki, Abdelkader El Mahdaouy, Ismail Berrada, and Ahmed Khoumsi. 2021a. Domain adaptation for Arabic cross-domain and cross-dialect sentiment analysis from contextualized word embedding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2824–2837, Online. Association for Computational Linguistics. find
  12. Abdellah El Mekki, Abdelkader El Mahdaouy, Kabil Essefar, Nabil El Mamoun, Ismail Berrada, and Ahmed Khoumsi. 2021b. BERT-based multi-task model for country and province level MSA and dialectal Arabic identification. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 271–275, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find
  13. Kabil Essefar, Abdellah El Mekki, Abdelkader El Mahdaouy, Nabil El Mamoun, and Ismail Berrada. 2021. CS-UM6P at SemEval-2021 task 7: Deep multi-task learning model for detecting and rating humor and offense. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1135–1140, Online. Association for Computational Linguistics. find
  14. Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. Idat at fire2019: Overview of the track on irony detection in arabic tweets. In Proceedings of the 11th Forum for Information Retrieval Evaluation, FIRE ’19, page 10–13, New York, NY, USA. Association for Computing Machinery. find
  15. Debanjan Ghosh, Avijit Vajpayee, and Smaranda Muresan. 2020. A report on the 2020 sarcasm detection shared task. In Proceedings of the Second Workshop on Figurative Language Processing, pages 1–11, Online. Association for Computational Linguistics. find our card
  16. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find our card
  17. John S. Leggitt and Raymond W. Gibbs. 2000. Emotional reactions to verbal irony. Discourse Processes, 29(1):1–24. ↗
  18. Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. CoRR, abs/1708.02002. ↗ our card
  19. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4238– 4243, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗ our card
  20. Augustus Odena. 2016. Semi-supervised learning with generative adversarial networks. find
  21. Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2854– 2859, Florence, Italy. Association for Computational Linguistics. ↗ our card
  22. Silviu Oprea and Walid Magdy. 2020. iSarcasm: A dataset of intended sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279–1289, Online. Association for Computational Linguistics. ↗ our card
  23. Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2020. Adapt or get left behind: Domain adaptation through BERT language model finetuning for aspect-target sentiment classification. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4933–4941, Marseille, France. European Language Resources Association. find
  24. Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. SemEval-2014 task 9: Sentiment analysis in Twitter. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 73–80, Dublin, Ireland. Association for Computational Linguistics. ↗ our card
  25. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card
  26. Deirdre Wilson. 2006. The pragmatics of verbal irony: Echo or pretence? Lingua, 116(10):1722–1743. Language in Mind: A Tribute to Neil Smith on the Occasion of his Retirement. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mahdaouy-2022-cs-um6p-semeval-2022-task/