Elamur Research⌕
☰
home / papers / 2021 / Domain Adaptation for Arabic Cross-Domain and Cross-Dialect Sentiment Analysis from Contextualized Word Embedding

Domain Adaptation for Arabic Cross-Domain and Cross-Dialect Sentiment Analysis from Contextualized Word Embedding

Paper · 2021 · Abdellah El Mekki, Abdelkader El Mahdaouy, Ismail Berrada et al.

Summary

Sentiment classifiers built on pre-trained language models lose accuracy when the test text comes from a different domain or dialect than the training data. For Arabic this is a serious problem. Dialects differ substantially from MSA and from one another, and labelling data for every dialect and domain is expensive. The authors frame the issue as unsupervised domain adaptation: labels exist only for the source, and the target supplies raw text alone. Their model uses AraBERT as a shared feature generator. Its [CLS] vector goes to a sentiment classifier and to the ALDA scheme, which joins adversarial alignment of the two domains with self-training on target pseudo-labels. A discriminator learns to correct noise in those pseudo-labels. They skip domain-adaptive MLM pre-training because unlabeled target text is scarce. They test three settings: cross-domain, cross-dialect, and both together, using coarse- and fine-grained dialect taxonomies and MSA-to-dialect transfer. Their method beats CORAL, MMD, DANN and older bag-of-words or pivot-based approaches on most datasets. On average it improves on zero-shot BERT by a relative 20.8%. This shows that labelled MSA data can be reused for dialects without new annotation.

Key points

  • Architecture: an AraBERT encoder shared by both domains, a softmax classifier on the [CLS] vector, and the ALDA losses, which are applied to the generator, the classifier and the discriminator that corrects pseudo-label noise.
  • Three adaptation scenarios are studied: cross-domain, cross-dialect, and cross-domain & cross-dialect. The authors describe this as the first such study for Arabic.
  • MSA→dialect transfer uses ArSAS as the source, with targets MSAC, TSAC, ASTD, AJGT, TweetSYR and AraSenti-Tweet. Accuracy compared with ZS-BERT:
TargetZS-BERTOurs
MSAC85.089.5
TSAC81.487.5
ASTD87.391.5
AJGT83.890.5
TweetSYR83.887.5
AraSenti-Tweet79.683.9
  • In the MSA→dialect setting, every DA method beats ZS-BERT by about 4.9% on average. All BERT-based methods (CORAL, MMD, DANN, ours) clearly outperform DANN_BOW, ADRL, HATN and PBLM.
  • Error analysis covers two kinds of cases:
    • Remaining failures are mostly very short dialectal phrases (idioms, insults, sarcasm) with vocabulary far from MSA.
    • Zero-shot BERT bias: it leans toward the source label prior, since ArSAS is 63% negative.

Interesting ideas

  • I1. Pseudo-labels from the target domain are not trusted directly; a discriminator learns a per-domain confusion matrix that corrects their noise before they enter the loss.
  • I2. Alignment should be class-wise rather than class-agnostic: matching overall feature distributions still confuses mixed-polarity reviews, while aligning per class fixes this.
  • I3. A cheap diagnostic is to check whether zero-shot predictions simply copy the source label prior, which was 63% negative here, before crediting the model with real transfer.
  • I4. Results for each target are averaged over several possible source dialects, so conclusions do not depend on one lucky source/target pair.
  • I5. The hardest residual errors were short idiomatic or sarcastic phrases with vocabulary far from the training variety.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver trees agreed by witness parsers could be weighted through a learned per-genre confusion matrix of label errors, instead of a fixed agreement threshold.
  • I2 (tree): When adapting gparse to a new genre or UD dialect conversion, compare distributions per dependency relation rather than globally, so that rare relations are not swamped.
  • I3 (learn): Add a negative control comparing the predicted label distribution on target data with the source prior, to flag models that only reproduce training priors.
  • I4 (learn): Evaluate cross-treebank transfer by averaging over all source treebanks for each target, giving a fairer picture of robustness.
  • I5 (frames): Mark very short, out-of-vocabulary idiomatic utterances as low-confidence for irony and speech-act decisions, and route them to explicit idiom or irony rules rather than default literal frames.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université Mohammed VI PolytechniqueMarrakesh · Morocco18 works · 171 citations🎓Université de SherbrookeSherbrooke · Canada4 works · 53 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Abdellah El Mekki, Abdelkader El Mahdaouy, Ismail Berrada and Ahmed Khoumsi. 2021. Domain Adaptation for Arabic Cross-Domain and Cross-Dialect Sentiment Analysis from Contextualized Word Embedding. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.naacl-main.226

@inproceedings{mekki2021domain,
  title = {{Domain Adaptation for Arabic Cross-Domain and Cross-Dialect Sentiment Analysis from Contextualized Word Embedding}},
  author = {Abdellah {El Mekki}, Abdelkader {El Mahdaouy}, Ismail Berrada and Ahmed Khoumsi},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.226},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.226}
}

References

56 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Houssem Abdellaoui and Mounir Zrigui. 2018. Using Tweets and Emojis to Build TEAD: an Arabic Dataset for Sentiment Analysis. Computación y Sistemas, 22:777 – 786. ↗
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Azadeh Hashemi, and El Moatez Billah Nagoudi. 2020. AraNet: A deep learning toolkit for Arabic social media. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 16–23, Marseille, France. European Language Resource Association. ↗
  3. Ibrahim Abu Farha and Walid Magdy. 2019. Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy. Association for Computational Linguistics.
  4. Mahmoud Al-Ayyoub, Abed Allah Khamaiseh, Yaser Jararweh, and Mohammed N. Al-Kabi. 2019. A comprehensive survey of arabic sentiment analysis. Information Processing & Management, 56(2):320 – 342. Advance Arabic Natural Language Processing (ANLP) and its Applications. ↗
  5. Ahmad Al Sallab, Hazem Hajj, Gilbert Badaro, Ramy Baly, Wassim El Hajj, and Khaled Bashir Shaban. 2015. Deep learning models for sentiment analysis in Arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 9–17, Beijing, China. Association for Computational Linguistics. ↗
  6. Nora Al-Twairesh, Hend Al-Khalifa, AbdulMalik Al-Salman, and Yousef Al-Ohali. 2017. Arasenti-tweet: A corpus for arabic sentiment analysis of saudi tweets. Procedia Computer Science, 117:63 – 72. Arabic Computational Linguistics. find
  7. Khaled Mohammad Alomari, Hatem M. ElSherif, and Khaled Shaalan. 2017. Arabic tweets sentimental analysis using machine learning. In Advances in Artificial Intelligence: From Theory to Practice, pages 602–610, Cham. Springer International Publishing. find
  8. Israa Alsarsour, Esraa Mohamed, Reem Suwaileh, and Tamer Elsayed. 2018. DART: A large dataset of dialectal Arabic tweets. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). Arabic Natural Language Processing (ANLP), pages 165–173, Doha, Qatar. Association for Computational Linguistics.
  9. Ramy Baly, Alaa Khaddaj, Hazem M. Hajj, Wassim El-Hajj, and Khaled Bashir Shaban. 2018. ArSentD-LEV: A multi-topic corpus for target-based sentiment analysis in arabic levantine tweets. In OSACT 3: The 3rd Workshop on Open-Source Arabic Corpora and Processing Tools, page 37. ↗
  10. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207, Florence, Italy. Association for Computational Linguistics.
  11. Minghao Chen, Shuai Zhao, Haifeng Liu, and Deng Cai. 2020. Adversarial-learned loss for domain adaptation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 3521–3528. AAAI Press. find
  12. Mohamed Aly and Amir Atiya. 2013. LABR: A large scale Arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498, Sofia, Bulgaria. Association for Computational Linguistics. ↗
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  14. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  15. Chunning Du, Haifeng Sun, Jingyu Wang, Qi Qi, and Jianxin Liao. 2020. Adversarial and domain-aware BERT for cross-domain sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4019–4028, Online. Association for Computational Linguistics. find
  16. Elnagar Ashraf and Einea Omar. 2016. Brad 1.0: Book reviews in arabic dataset. In 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA), pages 1–8. find
  17. Samhaa R. El-Beltagy. 2016. NileULex: A phrase and word level sentiment lexicon for Egyptian and Modern Standard Arabic. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2900–2905, Portorož, Slovenia. European Language Resources Association (ELRA).
  18. Gilbert Badaro, Ramy Baly, Hazem Hajj, Wassim El-Hajj, Khaled Bashir Shaban, Nizar Habash, Ahmad Al-Sallab, and Ali Hamdi. 2019. A survey of opinion mining in arabic: A comprehensive system perspective covering challenges and advances in tools, resources, models, applications, and visualizations. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 18(3). find
  19. Gilbert Badaro, Ramy Baly, Hazem Hajj, Nizar Habash, and Wassim El-Hajj. 2014. A large scale Arabic sentiment lexicon for Arabic opinion mining. In Proceedings of the EMNLP 2014 Workshop on AbdelRahim Elmadany, Hamdy Mubarak, and Walid Magdy. 2018. Arsas: An arabic speech-act and sentiment corpus of tweets. Proceedings of the 3rd Workshop on Open-source Arabic Corpora and Processing Tools OSACT’2018, 3:20.
  20. Ashraf Elnagar and Omar Einea. Brad 1.0: Book reviews in arabic dataset. In 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA), pages 1–8. IEEE. find
  21. Ashraf Elnagar, Yasmin S. Khalifa, and Anas Einea. 2018. Hotel Arabic-Reviews Dataset Construction for Sentiment Analysis Applications, pages 35–52. Springer International Publishing, Cham. find
  22. Hady ElSahar and Samhaa R. El-Beltagy. 2015. Building large arabic multi-domain resources for sentiment analysis. In Computational Linguistics and Intelligent Text Processing, pages 23–34, Cham. Springer International Publishing. find
  23. Alexander Erdmann, Nasser Zalmout, and Nizar Habash. 2018. Addressing noise in multidialectal word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 558–565. Association for Computational Linguistics. find
  24. Fadhl Eryani, Nizar Habash, Houda Bouamor, and Salam Khalifa. 2020. A spelling correction corpus for multiple arabic dialects. In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 4130–4138. European Language Resources Association.
  25. Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. J. Mach. Learn. Res., 17(1):2096–2030. ↗
  26. Hazem Hajj Wassim El-Hajj Gilbert Badaro, Hussein Jundi and Nizar Habash. 2018. Arsel: A large scale arabic sentiment and emotion lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France. European Language Resources Association (ELRA). find
  27. Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. 2012. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723–773. ↗
  28. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, Sara Hassan, Faisal Al-Shargi, Sakhar B. Alkhereyfy, Basma Abdulkareem, Ramy Eskander, Mohammad Salameh, and Hind Saddiki. 2018. Unified guidelines and resources for arabic dialect orthography. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA).
  29. Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. find
  30. Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4238–4248, Hong Kong, China. Association for Computational Linguistics. ↗
  31. Meftouh Karima, Harrat Salima, and Smaïli Kamel. 2018. PADIC: extension and new experiments. In 7th International Conference on Advanced Technologies ICAT, Antalya, Turkey. find
  32. Alaa Khaddaj, Hazem Hajj, and Wassim El-Hajj. 2019. Improved generalization of Arabic text classifiers. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 167–174, Florence, Italy. Association for Computational Linguistics.
  33. Almeman Khalid and Lee Mark. 2013. Automatic building of arabic multi dialect text corpora by bootstrapping dialect words. In 2013 1st International Conference on Communications, Signal Processing, and their Applications (ICCSPA), pages 1–6. find
  34. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems (NeurIPS). find
  35. Juntao Li, Ruidan He, Hai Ye, Hwee Tou Ng, Lidong Bing, and Rui Yan. 2020. Unsupervised domain adaptation of a pretrained cross-lingual language model. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3672–3678. International Joint Conferences on Artificial Intelligence Organization. Main track. find
  36. Zheng Li, Ying Wei, Yu Zhang, and Qiang Yang. 2018. Hierarchical attention transfer network for cross-domain sentiment classification. find
  37. Salima Medhaffar, Fethi Bougares, Yannick Estève, and Lamia Hadrich-Belguith. 2017. Sentiment analysis of Tunisian dialects: Linguistic ressources and experiments. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 55– 61, Valencia, Spain. Association for Computational Linguistics.
  38. Leila Moudjari, Karima Akli-Astouati, and Farah Benamara. 2020. An Algerian corpus and an annotation platform for opinion and emotion analysis. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1202–1210, Marseille, France. European Language Resources Association. find
  39. Abdul-Mageed Muhammad, Zhang Chiyu, Bouamor Houda, and Habash Nizar. 2020. NADI 2020: The first nuanced arabic dialect identification shared task. arXiv:2010.11334. ↗
  40. Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015. ASTD: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519, Lisbon, Portugal. Association for Computational Linguistics. ↗
  41. Oumaima Oueslati, Erik Cambria, Moez Ben HajHmida, and Habib Ounelli. 2020. A review of sentiment analysis research in arabic language. Future Generation Computer Systems, 112:408 – 430.
  42. Thuy-Trang Vu, Dinh Phung, and Gholamreza Haffari. 2020. Effective unsupervised domain adaptation with adversarially trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6163– 6173. Association for Computational Linguistics. find
  43. Ahmed Oussous, Fatima-Zahra Benjelloun, Ayoub Ait Lahcen, and Samir Belfkih. 2020. Asa: A framework for arabic sentiment analysis. Journal of Information Science, 46(4):544–559. ↗
  44. Hai Ye, Qingyu Tan, Ruidan He, Juntao Li, Hwee Tou Ng, and Lidong Bing. 2020. Feature adaptation of pre-trained language models across languages and domains with robust self-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 7386–7399. Association for Computational Linguistics. find
  45. Chatrine Qwaider, Stergios Chatzikyriakidis, and Simon Dobnik. 2019. Can Modern Standard Arabic approaches be used for Arabic dialects? sentiment analysis as a case study. In Proceedings of the 3rd Workshop on Arabic Corpus Linguistics, pages 40– 50, Cardiff, United Kingdom. Association for Computational Linguistics.
  46. Jianfei Yu and Jing Jiang. 2016. Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 236–246, Austin, Texas. Association for Computational Linguistics. find
  47. Alan Ramponi and Barbara Plank. 2020. Neural unsupervised domain adaptation in nlp—a survey. arXiv preprint arXiv:2006.00632. ↗
  48. Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41, Portland, Oregon, USA. Association for Computational Linguistics.
  49. Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2020. Adapt or get left behind: Domain adaptation through BERT language model finetuning for aspect-target sentiment classification. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4933–4941, Marseille, France. European Language Resources Association. find
  50. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  51. Houda Saadane and Nizar Habash. 2015. A conventional orthography for algerian arabic. In Proceedings of the Second Workshop on Arabic Natural Language Processing, ANLP@ACL 2015, Beijing, China, July 30, 2015, pages 69–79. Association for Computational Linguistics.
  52. Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He. 2015. Supervised representation learning: Transfer learning with deep autoencoders. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI’15, page 4119–4125. AAAI Press. find
  53. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  54. Yftah Ziser and Roi Reichart. 2018. Pivot based language modeling for improved neural domain adaptation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1241– 1251, New Orleans, Louisiana. Association for Computational Linguistics. find
  55. Mohammad Salameh, Saif Mohammad, and Svetlana Kiritchenko. 2015. Sentiment after translation: A case-study on Arabic social media posts. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 767–777, Denver, Colorado. Association for Computational Linguistics.
  56. Baochen Sun and Kate Saenko. 2016. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision – ECCV 2016 Workshops, pages 443–450, Cham. Springer International Publishing. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mekki-2021-domain-adaptation-arabic-cross-domain/