Elamur Research⌕
☰
home / papers / 2022 / AlexU-AL at SemEval-2022 Task 6: Detecting Sarcasm in Arabic Text Using Deep Learning Techniques

AlexU-AL at SemEval-2022 Task 6: Detecting Sarcasm in Arabic Text Using Deep Learning Techniques

Summary

The paper describes the AlexU-AL team's entry to the Arabic track of SemEval-2022 Task 6 (iSarcasmEval). The team worked on two sub-tasks. Sub-task A is binary sarcasm classification of single tweets. Sub-task C asks which of two texts with the same meaning is the sarcastic one. The training data is small: 3,102 tweets, about 24% of them sarcastic. It is unusual because the authors of the tweets labelled their own posts and wrote non-sarcastic rephrasings, and it mixes MSA with several dialects. To cope with the small, dialect-heavy data, the authors fine-tuned MARBERT, a BERT-Base-style model pretrained on roughly a billion Arabic tweets, with one linear classification layer on top. For sub-task C they gave both texts to the same classifier and picked the one with the higher sarcasm score. On their own held-out split the model reached an F1-sarcastic of 0.90, but on the official test set it scored only 0.5076. That result still placed 2nd of 32 teams in sub-task A. In sub-task C it placed 9th of 13. The paper shows that a tweet-domain, dialect-aware encoder is a strong baseline for Arabic sarcasm.

Key points

  • Data: 3,102 Arabic training tweets, of which 745 are sarcastic. Most are MSA (1,519) or Egypt/Nile (1,294), with few Gulf, Levant and Magreb examples. The sarcastic share differs sharply by dialect: very few MSA tweets are sarcastic.
  • Model: MARBERT via Hugging Face BertForSequenceClassification. Inputs were fixed at 64 tokens. Training ran for 4 epochs with learning rate 2e-06, batch size 32, an Adam weight-decay optimizer and cross-entropy loss. Data was split 80/10/10 for train, validation and test.
  • Metrics: sub-task A uses F1 on the sarcastic class only; sub-task C uses accuracy.
  • Official results: sub-task A F1-sarcastic 0.5076 (2nd of 32). Sub-task C accuracy 0.7450 and F-score 0.7442 (9th of 13).
  • There is a large gap between the internal test score (0.90) and the official score (0.5076). The authors leave error analysis for future work.

Interesting ideas

  • I1. The pairwise sub-task is solved without a dedicated pair model: one classifier scores each text separately and the higher score wins, which turns a single-text detector into a comparison judge for free. — For Mova (frames): the irony component in prag could score a sentence and a literal paraphrase of it and treat the score difference as explainable evidence of which reading is ironic.
  • I2. The dataset pairs each sarcastic text, labelled by its own author, with a literal rephrasing of the same intended message, giving clean minimal pairs that separate surface form from intent. — For Mova (learn): we could build ironic/literal paraphrase pairs as test items, checking that the frames and world states match while only the irony and speech-act flags differ.
  • I3. A near-perfect score on a small random in-house split collapsed on the real test set, a reminder that tiny imbalanced data with a random split can badly overstate quality. — For Mova (learn): our evaluation should keep a truly held-out set from a different source alongside negative controls, so an inflated internal number cannot hide overfitting.
  • I4. The encoder was chosen because its pretraining data matched the target genre (dialectal tweets), not just the target language, since MSA-heavy models fit the data less well. — For Mova (tree): when picking or pretraining an encoder for gparse, matching the genre of the target text (forum posts, tales, school texts) may matter as much as matching the language.
  • I5. Scoring only the F1 of the minority (sarcastic) class keeps a classifier from looking good by predicting the majority label everywhere. — For Mova (learn): rare phenomena such as irony or rare dependency relations should be reported with per-class F1 rather than overall accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

1 paper of our library cites it (of the 140884 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.125

@inproceedings{lotfy2022alexu,
  title = {{AlexU-AL at SemEval-2022 Task 6: Detecting Sarcasm in Arabic Text Using Deep Learning Techniques}},
  author = {Lotfy, Aya and Torki, Marwan and El-Makky, Nagwa},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.125},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.125}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Bamman and Noah Smith. 2015. Contextualized sarcasm detection on twitter. In Proceedings of the International AAAI Conference on Web and Social Media, volume 9, pages 574–577. find
  2. Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020. DAICT: A dialectal Arabic Farah Benamara, Cyril Grouin, Jihen Karoui, Véronique Moriceau, and Isabelle Robba. 2017. Analyse d’opinion et langage figuratif dans des tweets: présentation et résultats du défi fouille de textes deft2017. find
  3. John D Campbell and Albert N Katz. 2012. Are there necessary conditions for inducing a sense of sarcastic irony? Discourse Processes, 49(6):459–480. ↗
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  6. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 392–398, Istanbul, Turkey. European Language Resources Association (ELRA). ↗ our card
  7. Ahmed Hassan Yousef, Walaa Medhat, and Hoda Mohamed. 2014. Sentiment analysis algorithms and applications: A survey. Ain Shams Engineering Journal, 5. find
  8. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. arXiv preprint arXiv:1805.06413. ↗ our card
  9. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  10. Diana G Maynard and Mark A Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Lrec 2014 proceedings. ELRA. find our card
  11. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc. find
  12. Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. arXiv preprint arXiv:1910.11932. ↗ our card
  13. Silviu Vlad Oprea and Walid Magdy. 2020. The effect of sociocultural variables on sarcasm communication online. Proc. ACM Hum.-Comput. Interact., 4(CSCW1). find
  14. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  15. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In Proceedings of COLING 2014, the 25th international conference on computational linguistics: Technical papers, pages 213–223. find our card
  16. Aditya Joshi, Pushpak Bhattacharyya, Mark Carman, Jaya Saraswati, and Rajita Shukla. 2016. How do cultural differences impact the quality of sarcasm annotation?: A case study of indian annotators and american text. In Proceedings of the 10th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 95–99. find our card
  17. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714, Seattle, Washington, USA. Association for Computational Linguistics. find our card
  18. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext. zip: Compressing text classification models. arXiv preprint arXiv:1612.03651. ↗ our card
  19. Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. Semeval-2014 task 9: Sentiment analysis in twitter. pages 73–80. find our card
  20. Mikhail Khodak, Nikunj Saunshi, Yingyu Liang, Tengyu Ma, Brandon Stewart, and Sanjeev Arora. 2018. A la carte embedding: Cheap but effective induction of semantic feature vectors. arXiv preprint arXiv:1805.05388. ↗
  21. John Leggitt and Raymond Gibbs. 2000. Emotional reactions to verbal irony. Discourse Processes - DIS-COURSE PROCESS, 29:1–24. find
  22. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1–167. ↗
  23. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card
  24. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: Bert for finnish. arXiv preprint arXiv:1912.07076. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lotfy-2022-alexu-al-semeval-2022-task/