Elamur Research⌕
☰
home / papers / 2021 / AraBERT and Farasa Segmentation Based Approach For Sarcasm and Sentiment Detection in Arabic Tweets

AraBERT and Farasa Segmentation Based Approach For Sarcasm and Sentiment Detection in Arabic Tweets

Summary

This paper describes one team's system for the EACL WANLP-2021 Shared Task 2. The task has two subtasks on Arabic tweets: a binary decision on whether a tweet is sarcastic, and a three-way sentiment label (POS, NEG, NEU). The data is the ArSarcasm-v2 dataset, which the author splits 90:10 into training and validation parts.

The pipeline has two stages. First, a cleaning step removes markup, emoticons, repeated characters and extra spaces, adds spacing around digits and Latin characters, and swaps URLs, emails and user mentions for fixed placeholder tokens. Farasa segmentation is applied only for models pre-trained on segmented text. Second, the author fine-tunes six AraBERT variants and two AraELECTRA variants and picks the best one on validation for each subtask.

AraBERTv2-base was chosen for sarcasm and AraBERTv0.2-large for sentiment. The system placed seventh and fourth on the private leaderboard. The work matters as a clean, reproducible baseline for Arabic social-media classification, a language with limited sarcasm resources.

Key points

  • Data is ArSarcasm-v2: 12,548 training tweets and 3,000 test tweets. The local split is 11,293 train and 1,255 dev, and only 2,168 tweets in total are labelled sarcastic.
  • Eight models were compared: AraBERTv0.1/v0.2/v1/v2 in base and large sizes where available, plus the AraELECTRA generator and discriminator. All were loaded through Hugging Face and trained with lr 1e-5, max length 256, 10 epochs, and batch size 40 (base) or 4 (large).
  • The best validation weighted F1 scores were 85.55 for sarcasm (AraBERTv2-base) and 76.20 for sentiment (AraBERTv0.2-large). The AraELECTRA generator was the weakest model on both subtasks.
  • On the official test metrics, sarcastic-class F-score was 58.72 and F-PN (the macro F over positive and negative) was 72.55. The final ranks were 7th and 4th.
  • Newer AraBERT/AraELECTRA models share a 77GB pre-training corpus of about 8.6B words, drawn from OSCAR, OSIAN, Arabic Wikipedia, Assafir and the 1.5B words Arabic Corpus.

Interesting ideas

  • I1. Fine-tuning input is preprocessed to match each model's pre-training input, including applying Farasa segmentation only to models pre-trained on segmented text, so inputs match what each model saw in training. — For Mova (tokens): The tokenizer and lemmatizer in crate en should normalise input exactly as the treebank data used to train the tagger was normalised, with this match recorded as an explicit, checkable setting.
  • I2. URLs, emails and user mentions are replaced by fixed placeholder tokens rather than deleted, which keeps sentence structure while removing noise that carries no signal. — For Mova (tokens): Typed placeholder tokens for URLs, handles and emails would give the tagger and parser a stable noun-like slot, so web text does not break UD trees.
  • I3. Model selection used weighted F1 on a held-out split, yet the official target was the F-score of the rare sarcastic class, and validation (~85) and test (58.72) numbers diverge sharply. — For Mova (learn): Evaluation should always report minority-class metrics, such as irony or rare relations, next to weighted averages, so a model that wins on averages is not mistaken for one that handles the hard cases.
  • I4. Sarcasm is framed as a separate label alongside sentiment, which acknowledges that surface polarity and intended stance can disagree. — For Mova (frames): The irony module in crate prag could flag a mismatch between the literal polarity of a frame and the speaker's inferred stance, and give an explainable reason instead of a single opaque score.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

8 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2103.01679

@inproceedings{wadhawan2021arabert,
  title = {{AraBERT and Farasa Segmentation Based Approach For Sarcasm and Sentiment Detection in Arabic Tweets}},
  author = {Wadhawan, Anshul},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  eprint = {2103.01679},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.01679}
}

References

30 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020. DAICT: A dialectal Arabic irony corpus extracted from Twitter. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6265–6271, Marseille, France. European Language Resources Association. find our card
  2. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California. Association for Computational Linguistics. find our card
  3. Muhammad Abdul-Mageed, Mona Diab, and Mohammed Korayem. 2011. Subjectivity and sentiment analysis of Modern Standard Arabic. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 587–591, Portland, Oregon, USA. Association for Computational Linguistics. find
  4. Gavin Abercrombie and Dirk Hovy. 2016. Putting sarcasm detection into context: The effects of class imbalance and manual labelling on supervised machine classification of Twitter conversations. In Proceedings of the ACL 2016 Student Research Workshop, pages 107–113, Berlin, Germany. Association for Computational Linguistics. find our card
  5. Ibrahim Abu Farha and Walid Magdy. 2020. From Arabic sentiment analysis to sarcasm detection: The Ar-Sarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39, Marseille, France. European Language Resource Association. find
  6. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the wanlp 2021 shared task on sarcasm and sentiment detection in arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find our card
  7. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mário J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 167–177, Berlin, Germany. Association for Computational Linguistics. find our card
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. find our card
  9. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Araelectra: Pre-training text discriminators for arabic language understanding. find
  10. Francesco Barbieri, Francesco Ronzano, and Horacio Saggion. 2014a. Italian irony detection in twitter: a first approach. find
  11. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014b. Modelling sarcasm in Twitter, a novel approach. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 50–58, Baltimore, Maryland. Association for Computational Linguistics. find our card
  12. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn. Res., 3(null):1137–1155. find
  13. Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, and Inderjit Dhillon. 2020. Taming pretrained transformers for extreme multi-label text classification. find
  14. Kareem Darwish. 2014. Arabic information retrieval. Foundations and Trends® in Information Retrieval, 7:239–342. find
  15. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 392–398, Istanbul, Turkey. European Language Resources Association (ELRA). ↗ our card
  16. Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. Idat at fire2019: Overview of the track on irony detection in arabic tweets. In Proceedings of the 11th Forum for Information Retrieval Evaluation, FIRE ’19, page 10–13, New York, NY, USA. Association for Computing Machinery. find
  17. Debanjan Ghosh, Weiwei Guo, and Smaranda Muresan. 2015. Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1003– 1012, Lisbon, Portugal. Association for Computational Linguistics. find our card
  18. Alex Graves, Abdel rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. find
  19. Doaa Mohey El-Din Mohamed Hussein. 2018. A survey on sentiment analysis challenges. Journal of King Saud University - Engineering Sciences, 30(4):330–338. find
  20. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic sarcasm detection: A survey. ACM Comput. Surv., 50(5). find our card
  21. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762, Beijing, China. Association for Computational Linguistics. ↗ our card
  22. Aditya Joshi, Vaibhav Tripathi, Pushpak Bhattacharyya, and Mark J. Carman. 2016. Harnessing sequence labeling for sarcasm detection in dialogue from TV series ‘Friends’. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 146–155, Berlin, Germany. Association for Computational Linguistics. find our card
  23. Jihen Karoui, Farah Banamara Zitoune, and Véronique Moriceau. 2017. Soukhria: Towards an irony detection system for arabic in social media. Procedia Computer Science, 117:161–168. Arabic Computational Linguistics. find
  24. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS’12, page 1097–1105, Red Hook, NY, USA. Curran Associates Inc. find
  25. Bing Liu. 2012. Sentiment analysis and opinion mining. volume 5. find
  26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, page 3111–3119, Red Hook, NY, USA. Curran Associates Inc. find
  27. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM ’15, page 97–106, New York, NY, USA. Association for Computing Machinery. find
  28. Khaled Shaalan. 2010. Nizar y. habash, introduction to arabic natural language processing (synthesis lectures on human language technologies). Machine Translation, 24(3–4):285–289. find
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. find
  30. Wen-tau Yih, Kristina Toutanova, John C. Platt, and Christopher Meek. 2011. Learning discriminative projections for text similarity measures. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning, pages 247–256, Portland, Oregon, USA. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wadhawan-2021-arabert-farasa-segmentation-based-approach/