Elamur Research⌕
☰
home / papers / 2022 / Domain-Adapted BERT-based Models for Nuanced Arabic Dialect Identification and Tweet Sentiment Analysis

Domain-Adapted BERT-based Models for Nuanced Arabic Dialect Identification and Tweet Sentiment Analysis

Paper · 2022 · Giyaseddin Bayrak and Abdul Majeed Issifu

Summary

The paper describes a system for the NADI 2022 shared task, which had two parts: country-level Arabic Dialect Identification (ADID) and three-way Arabic Sentiment Analysis (ASA) on tweets. The dialect data covers 18 countries with about 20K training tweets. The sentiment set has 5,000 manually labelled tweets from 10 countries, mixing MSA and dialects. The authors fine-tune three Arabic pre-trained BERT models: AraBERT v0.2 Twitter-base (AraBERTtw), MARBERT and MARBERTv2. They use a fixed, stability-oriented fine-tuning recipe taken from Mosbach et al. They also compare training with and without AraBERT-style tweet normalization. MARBERTv2 with preprocessing gave the best dialect result: 33.89% macro F1 on Test-A and 25.54% averaged over both test sets. Plain MARBERT without preprocessing gave the best sentiment score, 74.29% Macro-F1-PN. The study argues two points. Models pre-trained on in-domain text (Arabic tweets) are the main driver of performance. The same normalization can help one task and hurt another, so it should be checked per task.

Key points

  • Fine-tuning recipe: 4 epochs, batch size 16, AdamW with lr 2e-5, weight decay 0.01, ε = 1e-6, gradient clipping and 10% warmup. The baseline used 5 epochs, batch 32, lr 5e-5, 500 warmup steps and ε = 1e-8, and it scored lower.
  • Preprocessing steps:
    • stripping Tashkeel and Tatweel;
    • mapping Hindi digits to Arabic digits;
    • padding non-Arabic digits with spaces;
    • cutting character repetitions down to two;
    • replacing "/" with "-" because the slash is not in AraBERT's vocabulary;
    • normalizing emojis instead of deleting them.
  • Test-B contains an unknown number k of countries. On Test-B the best model was MARBERT with preprocessing, at 17.51% macro F1.
  • Preprocessing generally helped dialect identification but tended to hurt sentiment. MARBERT generalized best overall, even though MARBERTv2 scores higher on the ARLUE benchmark.
  • Error analysis showed three things:
    • Countries with many training examples are classified better.
    • Neighbouring dialects get confused, for example Palestinian vs Jordanian and Saudi vs Emirati/Omani.
    • Some gold labels look wrong or ambiguous.

Interesting ideas

  • I1. Treat text normalization as a per-task switch and measure its effect on each task, since removing emojis and repetitions helped dialect ID but removed useful signal for sentiment.
  • I2. Shorten runs of repeated characters to two instead of one, which reduces sparsity but keeps a trace that emphasis was there.
  • I3. Map characters that the model vocabulary lacks (slash to dash) onto close in-vocabulary symbols before encoding, instead of letting them become unknown tokens.
  • I4. Compare the length distributions of correct and failed predictions (a KDE overlay) to see how input size relates to errors.
  • I5. Manually inspect confident errors to separate model mistakes from mislabelled or inherently ambiguous gold items.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Keep informal-English normalization (elongations, emoji, repeated punctuation) optional in crate en and record the raw form, so prag can still use it for irony and speech-act cues.
  • I2 (tokens): A lemma/normalization rule like "soooo" → "soo" could map onto the base lemma while flagging an intensity feature, instead of erasing it silently.
  • I3 (tree): The gparse input pipeline could carry a small, logged substitution table for out-of-vocabulary symbols, keeping offsets back to the original text for explainability.
  • I4 (learn): Plotting parser and reader failures against sentence length would be a cheap diagnostic, and could serve as a negative-control style check during evaluation.
  • I5 (learn): Cases where witness parsers agree but contradict the treebank are good candidates for gold-label review before they are counted as errors or fed into repair rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Marmara UniversityIstanbul · Türkiye2 works · 13 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Giyaseddin Bayrak and Abdul Majeed Issifu. 2022. Domain-Adapted BERT-based Models for Nuanced Arabic Dialect Identification and Tweet Sentiment Analysis. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.43

@inproceedings{bayrak2022domain,
  title = {{Domain-Adapted BERT-based Models for Nuanced Arabic Dialect Identification and Tweet Sentiment Analysis}},
  author = {Giyaseddin Bayrak and Abdul Majeed Issifu},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.43},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.43}
}

References

21 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020a. Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020b. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  3. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  4. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Seven Arabic Natural Language Processing Workshop (WANLP 2022).
  5. Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. arXiv preprint arXiv:2101.11038. ↗
  6. Onur Akça, Gıyaseddin Bayrak, Abdul Majeed Issifu, and Murat Can Ganz. 2022. Traditional machine learning and deep learning-based text classification for turkish law documents using transformers and domain adaptation. In 2022 International Conference on INnovations in Intelligent SysTems and Applications (INISTA), pages 1–6. IEEE. find
  7. Badr AlKhamissi, Mohamed Gabr, Muhammad El-Nokrashy, and Khaled Essam. 2021. Adapting marbert for improved arabic dialect identification: Submission to the nadi 2021 shared task. arXiv preprint arXiv:2103.01065. ↗
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  9. Gıyaseddin Bayrak, Muhammed Şakir Toprak, Murat Can Ganiz, Halife Kodaz, and Ural Koç. 2022. Deep learning-based brain hemorrhage detection in ct reports. In Challenges of Trustable AI and Added-Value on Health, pages 866–867. IOS Press. ↗
  10. Gökberk Çelkmasat, Muhammed Enes Aktürk, Yunus Emre Ertunç, Abdul Majeed Issifu, and Murat Can Ganz. 2022. Biomedical named entity recognition using transformers with bilstm+ crf and graph convolutional neural networks. In 2022 International Conference on INnovations in Intelligent SysTems and Applications (INISTA), pages 1–6. IEEE. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  12. Ismael Garrido-Muñoz, Arturo Montejo-Ráez, Fernando Martínez-Santiago, and L Alfonso Ureña-López. 2021. A survey on bias in deep nlp. Applied Sciences, 11(7):3184. ↗
  13. Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2019. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. arXiv preprint arXiv:1911.03437. ↗
  14. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. ↗
  15. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. arXiv preprint arXiv:2006.04884. ↗
  16. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  17. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  19. Anshul Wadhawan. 2021. Dialect identification in nuanced arabic tweets using farasa segmentation and arabert. arXiv preprint arXiv:2102.09749. ↗
  20. Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. Luke: deep contextualized entity representations with entity-aware self-attention. arXiv preprint arXiv:2010.01057. ↗
  21. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. Osian: Open source international arabic news corpus-preparation and integration into the clarin-infrastructure. In Proceedings of the fourth arabic natural language processing workshop, pages 175–182. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bayrak-2022-domain-adapted-bert-based-models/