Elamur Research⌕
☰
home / papers / 2021 / Multi-task Learning Using a Combination of Contextualised and Static Word Embeddings for Arabic Sarcasm Detection and Sentiment Analysis

Multi-task Learning Using a Combination of Contextualised and Static Word Embeddings for Arabic Sarcasm Detection and Sentiment Analysis

Summary

The paper addresses two tasks on Arabic tweets from the WANLP-2021 shared task: deciding whether a tweet is sarcastic and classifying its sentiment as negative, neutral or positive. The data are dialect-rich and class-imbalanced. The authors start from the observation that sarcasm typically inverts the apparent polarity of a message, so they train the two tasks together. Their system combines three components. The first is MARBERT, a BERT model pre-trained on dialectal tweets, fine-tuned once for each task; the two resulting vectors are joined into a multi-task representation. The second is static embeddings, either word-level AraVec or a character 3-gram FastText model (CE) trained on emotion-heavy tweets. The third is a CNN-LSTM network that processes the static vectors. The MARBERT and static representations are concatenated and passed to a softmax classifier. On ArSarcasm-v2 the system ranked first for sarcasm detection and 11th of 27 for sentiment. The work suggests that pairing related pragmatic tasks, and adding subword static vectors to contextual ones, helps with noisy dialectal text.

Key points

  • Dataset ArSarcasm-v2: 12,548 training and 3,000 test tweets.
    • Sarcasm is a minority class, with 2,168 sarcastic vs 10,380 non-sarcastic tweets in training.
    • Sentiment training labels: NEG 4,621, NEU 5,747, POS 2,180.
  • Contextual part: MARBERT, pre-trained on 6B tweets. Vectors are averaged over its 12 hidden layers after task-specific fine-tuning. The two task vectors (768 dimensions each) are concatenated into 1,536 dimensions.
  • Static part: AraVec (word-level) vs CE (character 3-grams, FastText, 10M tweets). In the CNN baseline, CE did better than AraVec (sarcasm F1 0.498 vs 0.486; F-PN 0.604 vs 0.534), so CE was used in the final system.
  • Results from five-fold cross-validation (no development set was available):
    • MTL-CNN-LSTM was best, with F1-sarcastic 0.632 and F-PN 0.713.
    • Single-task MARBERT came close, at 0.609 and 0.702.
  • Official test scores were F1-sarcastic 0.623 (rank 1st) and F-PN 0.701 (rank 11th). Preprocessing removed punctuation and diacritics, kept emojis, and normalised hamza and t marbouta letter forms.

Interesting ideas

  • I1. Sarcasm is treated as something that flips surface sentiment, so the sarcasm and sentiment tasks are learned jointly and each acts as a signal for the other. This is useful wherever two judgements are logically coupled. — For Mova (frames): the irony component in prag could output a literal polarity and an irony flag together, then derive the intended evaluation by an explicit flip rule that can be shown to the user.
  • I2. Character n-gram static embeddings handle dialectal spelling variants and out-of-vocabulary forms better than word-level vectors, because morphological variants end up close in vector space. — For Mova (tokens): a character 3-gram backoff could suggest lemma and tag candidates for misspelled, slang or unseen English words before the tagger falls back to a default guess.
  • I3. Contextual encoder vectors are simply concatenated with static subword vectors, which gives a cheap mix of context sensitivity and robustness for rare words. — For Mova (tree): gparse could append static char-n-gram vectors to the transformer token states, which may help attachment decisions for rare or noisy tokens.
  • I4. The headline metrics target what matters: F1 on the sarcastic minority class, and F-PN, which ignores neutral. The paper also reports a strong single-task baseline, which shows the multi-task gain on sentiment is small. — For Mova (learn): irony and speech-act evaluation should report minority-class F1 and always include the strongest single-component control, so that weak gains are not overclaimed.
  • I5. The authors note that stripping question and exclamation marks probably discarded sarcasm cues. Normalisation choices should therefore be checked against the target phenomenon. — For Mova (tokens): keep punctuation and emoji tokens as features in the tagger output so that later irony and speech-act detection can use them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

3 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 1 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alharbi2021multi,
  title = {{Multi-task Learning Using a Combination of Contextualised and Static Word Embeddings for Arabic Sarcasm Detection and Sentiment Analysis}},
  author = {Alharbi, Abdullah I. and Lee, Mark},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.39/}
}

References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ines Abbes, Wajdi Zaghouani, Omaima El-Hardlo, and Faten Ashour. 2020. DAICT: A dialectal Arabic irony corpus extracted from Twitter. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6265–6271, Marseille, France. European Language Resources Association. find our card
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. find our card
  3. Ibrahim Abu Farha and Walid Magdy. 2019. Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy. Association for Computational Linguistics. find
  4. Ibrahim Abu Farha and Walid Magdy. 2020. From Arabic sentiment analysis to sarcasm detection: The Ar-Sarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39, Marseille, France. European Language Resource Association. find
  5. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. find our card
  6. Israa Alghanmi, Luis Espinosa Anke, and Steven Schockaert. 2020. Combining BERT with static word embeddings for categorizing social media. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pages 28–33, Online. Association for Computational Linguistics. find
  7. Abdullah I. Alharbi and Mark Lee. 2020. Combining character and word embeddings for affect in Arabic informal social media microblogs. In Natural Language Processing and Information Systems, pages 213–224, Cham. Springer International Publishing. find
  8. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. ↗
  9. Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question Answering with Subgraph Embeddings. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 615–620. find
  10. M. Bouazizi and T. Otsuki Ohtsuki. 2016. A pattern-based approach for sarcasm detection on twitter. IEEE Access, 4:5477–5488. find
  11. Rich Caruana. 1997. Multitask Learning. Machine Learning. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis. Association for Computational Linguistics. find
  13. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1370–1380. find our card
  14. Rehab Duwairi and Mahmoud El-Orfali. 2014. A study of the effects of preprocessing strategies on sentiment analysis for Arabic text. Journal of Information Science, 40(4):501–513. find
  15. Bilal Ghanem, Jihen Karoui, F. Benamara, Véronique Moriceau, and P. Rosso. 2019. Idat at fire2019: Overview of the track on irony detection in Arabic tweets. Proceedings of the 11th Forum for Information Retrieval Evaluation. find
  16. Jihen Karoui, Farah Banamara Zitoune, and Véronique Moriceau. 2017. SOUKHRIA: Towards an irony detection system for Arabic in social media. Procedia Computer Science, 117:161–168. Arabic Computational Linguistics. find
  17. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. Association for Computational Linguistics. ↗ our card
  18. Chu-Cheng Lin, Waleed Ammar, Chris Dyer, and Lori Levin. 2015. Unsupervised POS Induction with Word Embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1311–1316. find
  19. N. Majumder, S. Poria, H. Peng, N. Chhaya, E. Cambria, and A. Gelbukh. 2019. Sentiment and sarcasm classification with multitask learning. IEEE Intelligent Systems, 34(3):38–43. find
  20. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. AraVec: A set of Arabic word embedding models for use in Arabic NLP. Procedia Computer Science, 117:256–265. find
  21. Xingyou Wang, Weijie Jiang, and Zhiyong Luo. 2016. Combination of convolutional and recurrent neural network for sentiment analysis of short texts. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2428–2437, Osaka, Japan. find
  22. The COLING 2016 Organizing Committee. find
  23. Chiyu Zhang and Muhammad Abdul-Mageed. 2019. Multi-task bidirectional transformer representations for irony detection. find
  24. Jiajun Zhang, Shujie Liu, Mu Li, Ming Zhou, and Chengqing Zong. 2014. Bilingually-constrained phrase embeddings for machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 111–121. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alharbi-2021-multi-task-learning-combination-contextualised/