Elamur Research⌕
☰
home / papers / 2021 / ArSarcasm Shared Task: An Ensemble BERT Model for SarcasmDetection in Arabic Tweets

ArSarcasm Shared Task: An Ensemble BERT Model for SarcasmDetection in Arabic Tweets

Summary

This shared-task paper tackles binary sarcasm detection in Arabic tweets (sub-task 1 of the WANLP 2021 ArSarcasm-v2 task). The training set has 12,548 dialectal and MSA tweets and is heavily skewed toward the non-sarcastic class.

Preprocessing

  • The authors removed hashtags, mentions, Latin letters, digits and some punctuation, after checking that these carried no useful signal.
  • They normalized Arabic orthography.
  • They replaced each emoji with an Arabic textual description.

Models

  • A fine-tuned AraBERTv02 classifier.
  • A Sentence-BERT encoder built on AraBERTv02 and trained with a contrastive loss on tweet pairs.
  • A "full model" that concatenates the outputs of the two and puts a small classifier on top.

The design is loosely motivated by the idea that people process literal and intended meaning separately. On the validation split, the combination beat each component and clearly beat the bag-of-words baselines. On the official test set it reached 59.89 F1 on the sarcastic class, ranking 5th of 27 teams. The work shows that pairing a token-level fine-tuned encoder with a sentence-level embedding helps on a hard, imbalanced pragmatic task.

Key points

  • Data: ArSarcasm-v2 training data contains 2,168 sarcastic and 10,380 non-sarcastic tweets across Egyptian, Gulf, Levantine, Maghrebi and MSA. A separate unlabeled test set has 3,000 tweets.
  • AraBERTv02 base: 12 layers, hidden size 768, 12 heads, pretrained on about 200M Arabic sentences. It was fine-tuned with learning rate 2e-05, batch 16, gradient accumulation 2, for 2 epochs.
  • Full model: the BERT output and the Sentence-BERT embedding are joined. The classifier has two linear layers, one with GELU and a final sigmoid, and was trained with SGD at learning rate 1e-04 for five epochs.
  • Validation F1 (80/20 split):
    • BOW+LR: 45.88
    • TF-IDF+LR: 49.66
    • Sentence-BERT embedding+LR: 57.88
    • Fine-tuned AraBERTv02: 61.06
    • Combination: 62.27, with higher precision (70.05) but 1.42 points lower accuracy
  • Test results: sarcastic-class F1 59.89, which is 2.36 below the winner. Macro-F1 was 72.51 and accuracy 78.30.

Interesting ideas

  • I1. Before discarding surface cues such as hashtag or mention counts, links and digits, the authors first checked whether they correlated with the label, which makes the cleaning step a justified decision rather than a habit. — For Mova (learn): candidate features or repair rules could each be accepted or rejected only after a logged correlation check against gold labels, keeping the pipeline explainable.
  • I2. Emojis were rewritten as textual descriptions in the target language, and the machine-translated descriptions were checked by hand, so non-verbal emotional cues become ordinary words a model can read. — For Mova (tokens): the tokenizer could map emoji to lemma-like description tokens with a POS, so the tagger and parser see real words and later stages can still use the emotional cue.
  • I3. Sarcasm was modeled as two views of one tweet: a token-level fine-tuned encoder for the surface reading and a whole-sentence embedding for the overall meaning, combined at the end. — For Mova (frames): an irony detector in crate prag could explicitly compare the literal frame polarity with a sentence-level stance signal and flag a mismatch, giving an explainable reason for an irony label.
  • I4. Class labels were turned into same/different sentence pairs to train a siamese encoder with contrastive loss, which multiplies training signal from a small labeled set. — For Mova (learn): small labeled sets, such as speech-act or irony tags, could be expanded into pairs to train similarity models that suggest silver labels for unlabeled sentences.
  • I5. Results were reported as minority-class F1 with variance across runs, and the authors noted that accuracy can drop while F1 rises on skewed data. — For Mova (learn): evaluations of rare phenomena like irony should report class-specific F1 with spread over runs, not accuracy alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bashmal2021arsarcasm,
  title = {{ArSarcasm Shared Task: An Ensemble BERT Model for SarcasmDetection in Arabic Tweets}},
  author = {Bashmal, Laila and AlZeer, Daliyah},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.40/}
}

References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Full Emoji List, v13.1, https://unicode.org/ emoji/charts/full-emoji-list.html , note = Accessed: 2021-02-01. ↗
  2. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the wanlp 2021 shared task on sarcasm and sentiment detection in arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find our card
  3. Ravinder Ahuja, Shantanu Bansal, Shuvam Prakash, Karthik Venkataraman, and Alisha Banga. 2018. Comparative study of different sarcasm detection algorithms based on behavioral approach. Procedia Computer Science, 143:411–418. 8th International Conference on Advances in Computing Communications (ICACC-2018). find
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. find our card
  5. University Press Cambridge. 2021. Cambridge Dictionary. Cambridge University Press. find
  6. Carol A. Capelli, Noreen Nakagawa, and Cary M. Madden. 1990. How children understand sarcasm: The role of context and intonation. Child Development, 61(6):1824–1841. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Megan L. Dress, Roger J. Kreuz, Kristen E. Link, and Gina M. Caucci. 2008. Regional variation in the use of sarcasm. Journal of Language and Social Psychology, 27(1):71–85. find
  9. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581–586, Portland, Oregon, USA. Association for Computational Linguistics. find our card
  10. Suzana Ilić, Edison Marrese-Taylor, Jorge Balazs, and Yutaka Matsuo. 2018. Deep contextualized word representations for detecting sarcasm and irony. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 2–7, Brussels, Belgium. Association for Computational Linguistics. ↗ our card
  11. Stacey L. Ivanko, Penny M. Pexman, and Kara M. Oli-neck. 2004. How sarcastic are you?: Individual differences and verbal irony. Journal of Language and Social Psychology, 23(3):244–271. find
  12. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762, Beijing, China. Association for Computational Linguistics. ↗ our card
  13. Jihen Karoui, Farah Banamara Zitoune, and Véronique Moriceau. 2017. Soukhria: Towards an irony detection system for arabic in social media. Procedia Computer Science, 117:161–168. Arabic Computational Linguistics. find
  14. Roger J. Kreuz and Gina M. Caucci. 2007. Lexical influences on the perception of sarcasm. FigLan-guages ’07, page 1–4, USA. Association for Computational Linguistics. find our card
  15. L. Liu, J. L. Priestley, Y. Zhou, H. E. Ray, and M. Han. 2019. A2text-net: A novel deep neural network for sarcasm detection. In 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI), pages 118–126. find
  16. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  17. Tomer R. Aharon-Peretz J. Shamay-Tsoory, S G. 2005. The neuroanatomical basis of understanding sarcasm and its relationship to social cognition. Neuropsychology, 19(3):288–300. find
  18. Jayashree Subramanian, Varun Sridharan, Kai Shu, and Huan Liu. 2019. Exploiting emojis for sarcasm detection. In Social, Cultural, and Behavioral Modeling - 12th International Conference, SBP-BRiMS 2019, Proceedings, pages 70–80. Springer Verlag. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bashmal-2021-arsarcasm-shared-task-ensemble-bert/