Elamur Research⌕
☰
home / papers / 2022 / LISACTeam at SemEval-2022 Task 6: A Transformer based Approach for Intended Sarcasm Detection in English Tweets

LISACTeam at SemEval-2022 Task 6: A Transformer based Approach for Intended Sarcasm Detection in English Tweets

Paper · 2022 · Abdessamad Benlahbib, Hamza Alami and Ahmed Alami

Summary

This paper describes the LISACTeam entry for SemEval-2022 Task 6 (iSarcasmEval), which targets intended sarcasm in English tweets. In this dataset, sarcasm labels come from the tweet authors themselves. Two things make the task hard: the training data is small (3468 tweets), and it is skewed, with only about a quarter of tweets sarcastic. The team took part in two sub-tasks:

  • Sub-Task A: binary detection of sarcastic tweets.
  • Sub-Task C: given a sarcastic tweet and the author's non-sarcastic rephrase, pick which one is sarcastic.

Their system fine-tunes BERTweet, a BERT-base-sized model pretrained on tweets with the RoBERTa procedure. Before training, they strip URLs and turn emojis into English words. For Sub-Task C they reuse the Sub-Task A classifier and choose the text with the higher sarcasm probability. They compare BERTweet with n-gram linear models, recurrent networks and other transformers. BERTweet comes first and RoBERTa base second, and the submission placed 8th on A and 5th on C. The work shows that a domain-matched pretrained encoder is a strong, simple baseline for detecting sarcasm on social media.

Key points

  • English training data: 3468 tweets (867 sarcastic, 2601 not). The Sub-Task A test set has 200 sarcastic and 1200 non-sarcastic tweets. The Sub-Task C test set has 200 sarcastic tweets paired with their rephrases.
  • Metrics: F1 on the sarcastic class for Sub-Task A, accuracy for Sub-Task C. The best English Sub-Task A system overall reached about 0.6052 F1-sarcastic.
  • BERTweet fine-tuning settings:
    • learning rate 1e-5;
    • 15 epochs;
    • max length 128;
    • batch size 32;
    • Adam optimizer with cross-entropy loss.
  • Models compared: The classical baselines were LinearSVC, Logistic Regression and MultinomialNB (all on 1–3-gram counts), the recurrent baselines were bidirectional LSTM and GRU networks, and the transformers were BERTweet base, RoBERTa base, uncased BERT base and DistilBERT. Each transformer was tested both with and without preprocessing.
  • Surprises in the results:
    • LinearSVC beat BERT base and DistilBERT on Sub-Task A.
    • On Sub-Task C, the classical n-gram models outperformed Bi-LSTM and Bi-GRU.

Interesting ideas

  • I1. A single-text classifier can solve a paired "which one is sarcastic?" task with no extra training, by scoring both texts and picking the higher probability.
  • I2. Using author-provided labels plus a literal rephrase of each sarcastic tweet gives pairs where the intended meaning is known, not guessed by annotators.
  • I3. Rewriting emojis into their English textual meaning keeps their emotional signal while making the input ordinary words a text model can handle.
  • I4. A strong n-gram LinearSVC baseline beat two transformer models, a reminder that cheap baselines must always be reported to judge whether a complex model really helps.
  • I5. Measuring F1 only on the minority (sarcastic) class stops a model from looking good by always predicting the majority label on skewed data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): an irony detector in prag could be checked the same way, by contrasting a sentence with its literal paraphrase and verifying that the irony score separates the two.
  • I2 (learn): such sarcastic/literal pairs could become evaluation items where the frames and world built for both texts should come out equivalent after irony is resolved.
  • I3 (tokens): the en tokenizer could map emojis to word glosses, so the tagger and parser see analyzable tokens instead of unknown symbols.
  • I4 (learn): every new irony or pragmatics component should be compared against a shallow n-gram control, acting as a negative control that shows structure-based reasoning adds value.
  • I5 (learn): evaluations of rare phenomena such as irony or implicature should report per-class F1 for the rare class, not overall accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université Ibn-TofailKenitra · Morocco3 works · 4 citations

Also on the byline: High-Tech Rabat-Agdal.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 works

Cite

Cite as Abdessamad Benlahbib, Hamza Alami and Ahmed Alami. 2022. LISACTeam at SemEval-2022 Task 6: A Transformer based Approach for Intended Sarcasm Detection in English Tweets. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.139

@inproceedings{benlahbib2022lisacteam,
  title = {{LISACTeam at SemEval-2022 Task 6: A Transformer based Approach for Intended Sarcasm Detection in English Tweets}},
  author = {Abdessamad Benlahbib, Hamza Alami and Ahmed Alami},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.139},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.139}
}

References

20 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics.
  2. Hamza Alami, Said Ouatik El Alaoui, Abdessamad Benlahbib, and Noureddine En-nahnahi. 2020. LISAC FSDM-USMBA team at SemEval-2020 task 12: Overcoming AraBERT’s pretrain-finetune discrepancy for Arabic offensive language identification. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2080–2085, Barcelona (online). International Committee for Computational Linguistics.
  3. Jorge A. Balazs and Juan D. Velásquez. 2016. Opinion mining and information fusion: A survey. Information Fusion, 27:95–110. ↗
  4. Bernhard E. Boser, Isabelle M. Guyon, and Vladimir N. Vapnik. 1992. A training algorithm for optimal margin classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, COLT ’92, page 144–152, New York, NY, USA. Association for Computing Machinery. find
  5. Mondher Bouazizi and Tomoaki Otsuki Ohtsuki. 2016. A pattern-based approach for sarcasm detection on twitter. IEEE Access, 4:5477–5488. ↗
  6. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111, Doha, Qatar. Association for Computational Linguistics.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Hai Ha Do, PWC Prasad, Angelika Maag, and Abeer Alsadoon. 2019. Deep learning for aspect-based sentiment analysis: A comparative review. Expert Systems with Applications, 118:272–299. ↗
  9. D.I. Hernández Farias and P. Rosso. 2017. Chapter 7 - irony, sarcasm, and sentiment analysis. In Federico Alberto Pozzi, Elisabetta Fersini, Enza Messina, and Bing Liu, editors, Sentiment Analysis in Social Networks, pages 113–128. Morgan Kaufmann, Boston. find
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735–1780. ↗
  11. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. ↗
  12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Sub-Task A - English F1-sarcastic F1-score Precision Recall Accuracy LinearSVC 0.3508 0.6055 0.6304 0.5957 0.7700 Logistic Regression 0.3076 0.6021 0.5950 0.6124 0.8200 MultinomialNB 0.2982 0.5924 0.5904 0.5948 0.8050 Bi-LSTM 0.2667 0.5575 0.5700 0.5536 0.7486 Bi-GRU 0.2358 0.5330 0.5463 0.5329 0.7221 DistilBERT (with preprocessing) 0.3070 0.5861 0.5975 0.5799 0.7743 DistilBERT (without preprocessing) 0.3267 0.5809 0.6162 0.5758 0.7350 BERT base (with preprocessing) 0.3388 0.5888 0.6258 0.5823 0.7407 BERT base (without preprocessing) 0.3087 0.5887 0.5983 0.5829 0.7793 RoBERTa base (with preprocessing) 0.3746 0.6454 0.6262 0.6823 0.8521 RoBERTa base (without preprocessing) 0.3984 0.6351 0.6642 0.6218 0.7886 0.4291 0.6513 0.6353 0.6896 0.7929 0.4334 0.6547 0.6917 0.6384 0.7964 Model BERTweet base (with preprocessing) (Official Submission) BERTweet base (without preprocessing) *Table 2: The obtained results of various models on Sub-Task A - English test set* Sub-Task C - English Model F1-score Precision Recall Accuracy LinearSVC 0.5392 0.5657 0.5952 0.5850 Logistic Regression 0.5996 0.6008 0.6004 0.6000 MultinomialNB 0.6033 0.6034 0.6033 0.6050 Bi-LSTM 0.4834 0.4835 0.4836 0.4850 Bi-GRU 0.5200 0.5233 0.5233 0.5200 RoBERTa base (with preprocessing) 0.7177 0.7172 0.7188 0.7200 RoBERTa base (without preprocessing) 0.7186 0.7186 0.7186 0.7200 0.7737 0.7735 0.7740 0.7750 0.7585 0.7581 0.7589 0.7600 BERTweet base (with preprocessing) (Official Submission) BERTweet base (without preprocessing) *Table 3: The obtained results of various models on Sub-Task C - English test set* Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4238– 4243, Reykjavik, Iceland. European Language Resources Association (ELRA).
  13. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9–14, Online. Association for Computational Linguistics.
  14. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12(85):2825–2830. ↗
  15. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on Czech and English Twitter. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 213–223, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
  16. Yafeng Ren, Donghong Ji, and Han Ren. 2018. Context-augmented convolutional neural networks for twitter sarcasm detection. Neurocomputing, 308:1–7. ↗
  17. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In NeurIPS EMC 2 Workshop. find
  18. Mohammad Tubishat, Norisma Idris, and Mohammad A.M. Abushariah. 2018. Implicit aspect extraction in sentiment analysis: Review, taxonomy, oppportunities, and open challenges. Information Processing & Management, 54(4):545–563. ↗
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  20. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/benlahbib-2022-lisacteam-semeval-2022-task-6/