Elamur Research⌕
☰
home / papers / 2021 / Combining Context-Free and Contextualized Representations for Arabic Sarcasm Detection and Sentiment Identification

Combining Context-Free and Contextualized Representations for Arabic Sarcasm Detection and Sentiment Identification

Summary

The paper tackles sarcasm detection and three-way sentiment classification of Arabic tweets, the two subtasks of the WANLP ArSarcasm shared task 2021. Tweets are short, noisy and dialect-heavy, and AraBERT was pre-trained mostly on Wikipedia and news, so it may handle informal or rare words poorly. The team (SPPU-AASM) builds a hybrid classifier with two branches. One is the frozen [CLS] vector from AraBERT v0.2. The other is a sentence vector learned by a CNN-BiLSTM over Mazajak skip-gram embeddings, which were trained on Twitter. The two vectors are concatenated, passed through dropout and fed to a softmax layer. Because the two sources tokenize text differently, the fusion happens at sentence level. On ArSarcasm v2 the hybrid beats plain AraBERT and earlier systems. On the official test set it reached about 0.61 F1 on the sarcastic class (2nd place) and about 0.71 F-PN for sentiment (10th place). The work suggests that cheap, domain-specific static vectors can partly replace costly domain re-pretraining of a language model.

Key points

  • Data: ArSarcasm v2 has 12,549 labelled tweets covering MSA, Gulf, Egyptian, Levantine and Maghrebi. The authors used an 80/20 stratified train/validation split (2,510 validation tweets) and a separate 3,000-tweet official test set.
  • Architecture:
    • AraBERT v0.2 (bert-base-arabertv02) provides a 768-dimensional [CLS] vector and is frozen.
    • Mazajak skip-gram vectors (from 250M tweets) feed a CNN with 256 filters and max-pooling, followed by a 128-unit BiLSTM.
    • The concatenated vector goes to dropout and then a softmax layer.
  • Training: Adam, learning rate 5e-5, batch size 16, early stopping, maximum length 100 tokens.
  • Preprocessing:
    • Hashtags, mentions, URLs, punctuation, diacritics and stopwords are removed.
    • Emojis are mapped to Arabic words through a custom dictionary.
    • The AraBERT preprocessor handles normalization.
  • Validation results, averaged over 5 runs:
    • Sarcastic-class F1 rose from 0.52 (AraBERT) to 0.62.
    • F-PN rose from 0.67 to 0.71.
    • Recall gains were the largest.
  • Official test results were F1-sarcastic 0.614 and F-PN 0.707. The abstract instead reports 0.62 and 0.715.

Interesting ideas

  • I1. When a subword model and a word-level resource segment text differently, fuse their outputs as whole-sentence vectors rather than forcing a token alignment; this sidesteps tokenization mismatch entirely. — For Mova (tree): gparse could take a sentence- or word-level feature from domain-specific static vectors alongside transformer states, avoiding fragile subword-to-word alignment code.
  • I2. Static embeddings trained on the target domain's text can cover rare, slangy or dialectal words that a general pretrained encoder handles badly, at a fraction of the cost of re-pretraining. — For Mova (tokens): the tagger and lemmatizer could fall back on domain-trained static vectors for out-of-vocabulary or informal forms, such as social media text in fables or school texts.
  • I3. The pretrained encoder stays frozen while only the small static-vector branch and classifier are trained, which keeps training cheap and stable on a modest dataset. — For Mova (learn): new specialised heads, such as an irony or speech-act classifier, could be trained on a frozen gparse encoder without disturbing the parser itself.
  • I4. Emojis are turned into words of the target language before modelling, so their emotional signal enters the same lexical pipeline as ordinary text. — For Mova (tokens): an emoji-to-lemma table would let the tokenizer emit proper lemmas that downstream frames and sentiment cues can reason about explicitly.
  • I5. The authors note that sarcasm often hides negative sentiment behind literal wording, and that gains in recall on the minority sarcastic class were the clearest sign of improvement. — For Mova (frames): an irony detector in crate prag should be evaluated mainly by minority-class recall and F1 rather than accuracy, with results averaged over several runs as the paper does.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

3 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2103.05683

@inproceedings{hengle2021combining,
  title = {{Combining Context-Free and Contextualized Representations for Arabic Sarcasm Detection and Sentiment Identification}},
  author = {Hengle, Amey and Kshirsagar, Atharva and Desai, Shaily and Marathe, Manisha},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  eprint = {2103.05683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.05683}
}

References

29 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020. Daict: A dialectal arabic irony corpus extracted from twitter. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 6265–6271. find our card
  2. Hamdy Mubarak AbdelRahim Elmadany and Walid Magdy. 2018. Arsas: An arabic speech-act and sentiment corpus of tweets. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). find
  3. Ibrahim Abu Farha and Walid Magdy. 2019. Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy. Association for Computational Linguistics. find
  4. Ibrahim Abu Farha and Walid Magdy. 2020. From Arabic sentiment analysis to sarcasm detection: The Ar-Sarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39, Marseille, France. European Language Resource Association. pages 165–173, Doha, Qatar. Association for Computational Linguistics. find
  5. Ramy Baly, Gilbert Badaro, Georges El-Khoury, Rawan Moukalled, Rita Aoun, Hazem Hajj, Wassim El-Hajj, Nizar Habash, and Khaled Shaban. 2017. A characterization study of Arabic Twitter data with a benchmarking for state-of-the-art opinion mining models. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 110–118, Valencia, Spain. Association for Computational Linguistics. find
  6. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606. ↗
  7. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the wanlp 2021 shared task on sarcasm and sentiment detection in arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find our card
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ↗
  9. Mahmoud Al-Ayyoub, Abed Allah Khamaiseh, Yaser Jararweh, and Mohammed N Al-Kabi. 2019. A comprehensive survey of arabic sentiment analysis. Information processing & management, 56(2):320– 342. find
  10. Samhaa R. El-Beltagy. 2016. NileULex: A phrase and word level sentiment lexicon for Egyptian and Modern Standard Arabic. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2900–2905, Portorož, Slovenia. European Language Resources Association (ELRA). find
  11. Ahmad Al Sallab, Hazem Hajj, Gilbert Badaro, Ramy Baly, Wassim El Hajj, and Khaled Bashir Shaban. 2015. Deep learning models for sentiment analysis in Arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 9– 17, Beijing, China. Association for Computational Linguistics. find
  12. Mohammad Al-Smadi, Omar Qawasmeh, Mahmoud Al-Ayyoub, Y. Jararweh, and Brij Gupta. 2018. Deep recurrent neural network vs. support vector machine for aspect-based sentiment analysis of arabic hotels’ reviews. J. Comput. Sci., 27:386–393. find
  13. Abdulaziz M. Alayba, Vasile Palade, Matthew England, and Rahat Iqbal. 2018. A combined cnn and lstm model for arabic sentiment analysis. Machine Learning and Knowledge Extraction, page 179–191. find
  14. Israa Alghanmi, Luis Espinosa Anke, and Steven Schockaert. 2020. Combining BERT with static word embeddings for categorizing social media. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pages 28–33, Online. Association for Computational Linguistics. find
  15. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗ our card
  16. Gilbert Badaro, Ramy Baly, Hazem Hajj, Nizar Habash, and Wassim El-Hajj. 2014. A large scale Arabic sentiment lexicon for Arabic opinion mining. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. Idat at fire2019: Overview of the track on irony detection in arabic tweets. In Proceedings of the 11th Forum for Information Retrieval Evaluation, pages 10–13. find
  17. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural networks, 18(5-6):602–610. find
  18. M. Heikal, Marwan Torki, and N. El-Makky. 2018. Sentiment analysis of arabic tweets using deep learning. In ACLING. find
  19. Ibrahim Kaibi, Hassan Satori, et al. 2020. Sentiment analysis approach based on combination of word embedding techniques. In Embedded Systems and Artificial Intelligence, pages 805–813. Springer. find
  20. Jihen Karoui, Farah Banamara Zitoune, and Veronique Moriceau. 2017. Soukhria: Towards an irony detection system for arabic in social media. Procedia Computer Science, 117:161–168. ↗
  21. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. find
  22. Svetlana Kiritchenko, Saif Mohammad, and Mohammad Salameh. 2016. SemEval-2016 task 7: Determining sentiment intensity of English and Arabic phrases. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 42–51, San Diego, California. Association for Computational Linguistics. find
  23. Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. Kagnet: Knowledge-aware graph networks for commonsense reasoning. find
  24. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. find
  25. Nicole Peinelt, Dong Nguyen, and Maria Liakata. 2020. tBERT: Topic models and BERT joining forces for semantic similarity detection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7047–7055, Online. Association for Computational Linguistics. find
  26. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. E-BERT: Efficient-yet-effective entity embeddings for BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 803–818, Online. Association for Computational Linguistics. find
  27. Dina Said, Nayer M Wanas, Nevin M Darwish, and Nadia Hegazy. 2009. A study of text preprocessing tools for arabic text categorization. In The second international conference on Arabic language, pages 230–236. find
  28. Timo Schick and Hinrich Schütze. 2019. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. find
  29. Abu Bakr Soliman, Kareem Eissa, and Samhaa R. El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. Arabic Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hengle-2021-combining-context-free-contextualized-representations/