Elamur Research⌕
☰
home / papers / 2022 / Plumeria at SemEval-2022 Task 6: Robust Approaches for Sarcasm Detection for English and Arabic Using Transformers and Data Augmentation

Plumeria at SemEval-2022 Task 6: Robust Approaches for Sarcasm Detection for English and Arabic Using Transformers and Data Augmentation

Summary

The paper describes team Plumeria's entry to SemEval-2022 Task 6 (iSarcasm). The task asks systems to detect sarcasm in English and Arabic tweets, which matters because sarcasm inverts literal meaning and misleads sentiment and opinion systems. It has five subtasks: binary sarcasm detection (A, both languages), multi-label classification into six ironic-speech categories (B, English only), and choosing which of two texts is sarcastic, a tweet or its literal rephrase (C, both languages). The authors fine-tuned transformers (BERTweet, ELECTRA, SQuAD-tuned BERT, mBERT, CAMeLBERT-Mix) and also tried hierarchical models that place a BiLSTM, with or without attention, on top of BERT. Their main lever was the data. They merged the official sets with external corpora, deliberately skewed class ratios toward the scored class, and generated extra instances through GloVe-based word substitution or plain repetition. They also compared four preprocessing levels. The team placed fourth in three subtasks and sixth in another, and the authors argue this consistency shows their approach is robust. Stand-alone fine-tuned transformers with light, tweet-aware normalization worked best.

Key points

  • Official training data: 3468 English and 3102 Arabic tweets, about a quarter of them sarcastic. Each test set has 1400 tweets with roughly 14% sarcastic.
  • External data: tweets re-downloaded via the Twitter API (2841 train, 713 test), the SemEval-2018 Task 3 set (3834, balanced), and ArSarcasm-v2 (12548 train, 3000 test).
  • Word-embedding augmentation uses Gensim with glove-twitter-100. It swaps up to four keywords per tweet for one of their three nearest neighbours.
  • Four preprocessing types were compared:
    • Type I: raw text.
    • Type II: emoji→text, HTTPURL, @USER.
    • Type III: Type II plus normalization of smileys, punctuation runs, character elongations and contractions.
    • Type IV: Type III plus lemmatization/stemming and stop-word removal. Type II was generally the strongest.
  • Final results:
    • Subtask B: macro-F1 0.0778, rank 6, with BERTweet-large.
    • Subtask C English: accuracy 0.79, rank 4, with BERT large cased fine-tuned on SQuAD.
    • Subtask C Arabic: accuracy 0.87, rank 4, with CAMeLBERT-Mix trained on repetition-augmented data.

Interesting ideas

  • I1. For the pairwise task, half the training pairs had their two texts swapped and the label flipped, so the model cannot exploit position and must judge content. — For Mova (frames): when training the irony component on paired inputs (ironic vs literal), randomly swapping order with label flips would stop the scorer from learning a slot bias instead of an irony cue.
  • I2. The dataset of sarcastic tweets paired with human literal rephrases is effectively a supervision signal for intended versus said meaning. — For Mova (frames): parse both texts into frames and diff them; the state or evaluation whose polarity flips between the two is an explainable marker of where the irony lives.
  • I3. Instead of one class-balancing choice, they built a graded series of training sets (B0–B9) with sarcastic share stepping from 66% to 51%, then picked the ratio that best served a minority-class F1. — For Mova (learn): sweeping the mix of silver data along a single axis gives a clean, controlled way to see how composition, rather than volume, moves a target metric.
  • I4. Preprocessing works best when it mirrors the pretraining normalization (placeholder URL and mention tokens), while aggressive stemming and stop-word removal tended to hurt. — For Mova (tokens): map URLs, mentions and emojis in social-media input to fixed placeholder tokens before tagging, but keep function words and full forms, since the parser needs them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2203.04111 · arXiv 2203.04111

@misc{nigam2022plumeria,
  title = {{Plumeria at SemEval-2022 Task 6: Robust Approaches for Sarcasm Detection for English and Arabic Using Transformers and Data Augmentation}},
  author = {Nigam, Shubham Kumar and Shaheen, Mosab},
  year = {2022},
  doi = {10.48550/arxiv.2203.04111},
  eprint = {2203.04111},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2203.04111}
}

References

17 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. find our card
  2. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 296–305, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find our card
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for Arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. find our card
  4. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics. find our card
  5. L Bing. 2012. Sentiment analysis and opinion mining (synthesis lectures on human language technologies). University of Illinois: Chicago, IL, USA. find
  6. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR. find
  7. Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. Idat at fire2019: Overview of the track on irony detection in Arabic tweets. In Proceedings of the 11th Forum for Information Retrieval Evaluation, pages 10–13. ↗
  8. Debanjan Ghosh, Avijit Vajpayee, and Smaranda Muresan. 2020. A report on the 2020 sarcasm detection shared task. In Proceedings of the Second Workshop on Figurative Language Processing, pages 1– 11, Online. Association for Computational Linguistics. find our card
  9. Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation. Association for Computational Linguistics. find our card
  10. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics. find our card
  11. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315– 7330, Online. Association for Computational Linguistics. ↗ our card
  12. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4238–4243, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗ our card
  13. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9–14. find our card
  14. Silviu Oprea and Walid Magdy. 2020. iSarcasm: A dataset of intended sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279–1289, Online. Association for Computational Linguistics. ↗ our card
  15. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784– 789, Melbourne, Australia. Association for Computational Linguistics. ↗ our card
  16. Sara Rosenthal, Alan Ritter, Preslav Nakov, and Veselin Stoyanov. 2014. SemEval-2014 task 9: Sentiment analysis in Twitter. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 73–80, Dublin, Ireland. Association for Computational Linguistics. ↗ our card
  17. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nigam-2022-plumeria-semeval-2022-task-6/