Elamur Research⌕
☰
home / papers / 2022 / Plumeria at SemEval-2022 Task 6: Sarcasm Detection for English and Arabic Using Transformers and Data Augmentation

Plumeria at SemEval-2022 Task 6: Sarcasm Detection for English and Arabic Using Transformers and Data Augmentation

Summary

The paper reports team Plumeria's systems for SemEval-2022 Task 6 (iSarcasm), which covers sarcasm in English and Arabic tweets. Subtask A is binary detection, in both languages. Subtask B is English-only multi-label tagging over six ironic-speech categories. Subtask C asks a system to pick the sarcastic text from a pair made of a tweet and its literal rephrase, in both languages.

The core models are fine-tuned transformers: BERTweet-large for English and CAMeLBERT-Mix for Arabic. Some variants add a BiLSTM layer on top, optionally followed by dot-product attention. Because training data was small and imbalanced, the authors built enlarged datasets in three ways:

  • skewing them toward the minority class with external corpora,
  • swapping keywords for GloVe neighbours,
  • simply duplicating instances.

They also compared four levels of tweet preprocessing.

Plain fine-tuned transformers with Type II preprocessing worked best. The team placed fourth in three subtasks, sixth in one and tenth in one. The authors argue this evenness across subtasks shows the methods are robust. Error analysis shows that class-skewed training inflated recall at a heavy cost in precision.

Key points

  • External data came from three sources: tweets re-fetched through the Twitter API (2841 train, 713 test), the SemEval-2018 Task 3 data, and ArSarcasm-v2 for Arabic.
  • Embedding augmentation used Gensim with glove-twitter-100. In each copied tweet, up to four keywords were replaced by one of their top-3 most similar words.
  • Four preprocessing types were tested:
    • Type I: raw text.
    • Type II: emoji converted to text, URLs to "HTTPURL", mentions to "@USER", matching BERTweet pretraining.
    • Type III: adds normalisation of smileys, repeated characters and contractions.
    • Type IV: adds stemming and stop-word removal (WordNet lemmatizer for English, ISRI stemmer for Arabic).
  • English subtask A used the B4 training set (59% sarcastic). On test it reached TP=155, FP=295, FN=45, giving precision 34.44 and recall 77.5. For Arabic, precision was 26.69 and recall 85.
  • Subtask C was the strongest result, e.g. 0.87 accuracy and 4th place in Arabic. In subtask B, the model never correctly predicted understatement or overstatement, which were too rare in the data.

Interesting ideas

  • I1. Oversampling the target class to optimise that class's F1 moved the decision point sharply: recall rose while precision collapsed, a useful warning about shaping training data to fit a metric. — For Mova (learn): when silver data for rare phenomena such as irony is enriched, always score against a held-out set with the natural class distribution, used as a negative control on precision.
  • I2. The pairwise setup (ironic original vs. same-meaning literal rephrase) proved far easier than judging a single text in isolation, because the contrast exposes what sarcasm adds. — For Mova (frames): test the irony and implicature layer in prag on such minimal pairs, checking that the who-did-what frames match while the speech-act or irony annotation differs.
  • I3. Normalising URLs, mentions and emoji to the same placeholders the model saw in pretraining mattered. Mismatched emoji handling visibly hurt the Arabic model. — For Mova (tokens): define one canonical placeholder scheme for social-media artefacts in crate en and apply it identically at training and inference.
  • I4. The error analysis sorts cases by explainable cues: exclamation marks, very short tweets, clashing emotions, emoji, misspellings, and cases needing world knowledge. — For Mova (frames): use a sentiment clash between an evaluative predicate and the described situation as an explicit irony signal. Route the knowledge-dependent cases to world facts instead of guessing.
  • I5. Synthesising new examples by swapping keywords for embedding neighbours is a cheap way to grow small training sets, though it can shift meaning. — For Mova (learn): such paraphrase-like variants could stress-test whether frame extraction stays stable under lexical substitution, with witness parsers checking that the UD tree is unchanged.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.130

@inproceedings{shaheen2022plumeria,
  title = {{Plumeria at SemEval-2022 Task 6: Sarcasm Detection for English and Arabic Using Transformers and Data Augmentation}},
  author = {Shaheen, Mosab and Nigam, Shubham Kumar},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.130},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.130}
}

References

17 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSar-casmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. find
  2. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315– 7330, Online. Association for Computational Linguistics. ↗ our card
  3. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 296–305, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find our card
  4. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4238– 4243, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗ our card
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for Arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. find our card
  6. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9–14. find our card
  7. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics. find our card
  8. L Bing. 2012. Sentiment analysis and opinion mining (synthesis lectures on human language technologies). University of Illinois: Chicago, IL, USA. find
  9. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR. find
  10. Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. Idat at fire2019: Overview of the track on irony detection in Arabic tweets. In Proceedings of the 11th Forum for Information Retrieval Evaluation, pages 10–13. ↗
  11. Debanjan Ghosh, Avijit Vajpayee, and Smaranda Muresan. 2020. A report on the 2020 sarcasm detection shared task. In Proceedings of the Second Workshop on Figurative Language Processing, pages 1–11, Online. Association for Computational Linguistics. find our card
  12. Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find our card
  13. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics. find our card
  14. Silviu Oprea and Walid Magdy. 2020. iSarcasm: A dataset of intended sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279–1289, Online. Association for Computational Linguistics. ↗ our card
  15. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics. ↗ our card
  16. Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. SemEval-2014 task 9: Sentiment analysis in Twitter. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 73–80, Dublin, Ireland. Association for Computational Linguistics. ↗ our card
  17. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shaheen-2022-plumeria-semeval-2022-task-6/