Elamur Research⌕
☰
home / papers / 2022 / Dialect & Sentiment Identification in Nuanced Arabic Tweets Using an Ensemble of Prompt-based, Fine-tuned, and Multitask BERT-Based Models

Dialect & Sentiment Identification in Nuanced Arabic Tweets Using an Ensemble of Prompt-based, Fine-tuned, and Multitask BERT-Based Models

Paper · 2022 · Reem Abdel-Salam

Summary

This paper describes a first-place system for the NADI 2022 shared task, which covers two problems in Arabic tweets. Subtask 1 is country-level dialect identification across 18 Arab countries, with heavily skewed class sizes. Subtask 2 is three-way sentiment classification over tweets in ten dialects. The author tried several families of BERT-based Arabic encoders, including MARBERT, AraBERT, QARiB, AraELECTRA, CAMeLBERT and GigaBERT, adapted in three ways:

  • standard fine-tuning with attention or weighted-pooling heads;
  • multitask learning that predicts dialect and sentiment together;
  • several prompt-tuning variants: P-tuning via OpenPrompt, P-tuning v2 with or without an LSTM prompt encoder, and prefix prompting.

The final submissions combine these models through dev-optimized weighted ensembles or voting. The system ranked first in both subtasks, with an overall macro-F1 of 27.06 on dialects and an F1-PN of 75.15 on sentiment. Prompt-based models generally beat plain fine-tuning and the multitask model. The work matters because it shows that prompt formulations and loss-function mixing help in a noisy, low-resource, imbalanced setting.

Key points

  • Subtask 1 data: 20,398 training and 4,871 dev tweets over 18 country labels, strongly imbalanced. Subtask 2 has positive, negative and neutral labels and is roughly balanced.
  • Official metrics: macro-F1 for dialects. For sentiment, F1-PN is used, which is macro-F1 computed only over the positive and negative classes.
  • Subtask 1 results: the dev-optimized weighted ensemble scored best on both test sets (about 36.48 on Test A and 17.6 on Test B). MARBERT with a prompt was a very close second. The overall score was 27.06.
  • Subtask 2 results: the best submission (75.155 F1-PN) was a weighted ensemble of fine-tuned, multitask and prompt models. In another seven-model ensemble, the weight search simply returned uniform weights.
  • Imbalance: oversampling, undersampling and balanced batch sampling did not help. In error analysis, frequent dialects such as Egyptian and Saudi absorbed many predictions. Palestine, Qatar, Bahrain and UAE were almost never recognized correctly.

Interesting ideas

  • I1. When two datasets each carry labels for only one task, a fine-tuned single-task model can fill in the missing labels. The merged, fully labeled set then trains one multitask model.
  • I2. Ensemble member weights are tuned on the dev set, and members that end up with zero weight are dropped as redundant.
  • I3. For prompt-based sentiment, letting the model output only three fixed label words worked better than allowing synonyms.
  • I4. The sentiment metric deliberately ignores the neutral class, so a system cannot score well by defaulting to the uninformative label.
  • I5. The confusion matrix showed that majority classes swallow the rare ones, which summary scores hide.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Treebanks annotated only for syntax and corpora annotated only for semantic roles could be completed with silver labels from the existing models, so one joint model sees both signals.
  • I2 (learn): Weighting the witness parsers this way would show which witnesses add nothing to silver-data agreement. This should be checked against negative controls so the weights do not overfit the dev set.
  • I3 (frames): Speech-act or irony decisions could be scored over a small closed inventory of label words rather than open-ended descriptions, which keeps each decision explainable and stable.
  • I4 (frames): Implicature and irony detection could be evaluated only on the non-default classes, so a reader that always says "literal" earns no credit.
  • I5 (tree): Reporting per-relation confusion for the parsers would reveal rare UD relations collapsing into frequent ones like obl or dep, especially after dialect conversion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cairo UniversityGiza · Egypt42 works · 1,350 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Reem Abdel-Salam. 2022. Dialect & Sentiment Identification in Nuanced Arabic Tweets Using an Ensemble of Prompt-based, Fine-tuned, and Multitask BERT-Based Models. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.48

@inproceedings{abdelsalam2022dialect,
  title = {{Dialect \& Sentiment Identification in Nuanced Arabic Tweets Using an Ensemble of Prompt-based, Fine-tuned, and Multitask BERT-Based Models}},
  author = {Reem Abdel-Salam},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.48},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.48}
}

References

18 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021a. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  4. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021b. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  5. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Seven Arabic Natural Language Processing Workshop (WANLP 2022).
  6. Badr AlKhamissi, Mohamed Gabr, Muhammad El-Nokrashy, and Khaled Essam. 2021. Adapting marbert for improved arabic dialect identification: Submission to the nadi 2021 shared task. arXiv preprint arXiv:2103.01065. ↗
  7. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics. ↗
  8. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. Gigabert: Zero-shot transfer learning from english to arabic. In Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP). find
  9. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691. ↗
  10. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. ↗
  11. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988. ↗
  12. Hamada Nayel, Ahmed Hassan, Mahmoud Sobhi, and Ahmed El-Sawy. 2021. Machine learning-based approach for arabic dialect identification. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 287–290.
  13. Samia Touileb. 2020. Ltg-st at nadi shared task 1: Arabic dialect identification using a stacking classifier. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 313–319.
  14. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021a. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  15. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021b. AraGPT2: Pre-trained transformer for Arabic language generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196–207, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  16. Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Hai-Tao Zheng, and Maosong Sun. 2021. Openprompt: An open-source framework for prompt-learning. arXiv preprint arXiv:2111.01998. ↗
  17. Abdellah El Mekki, Abdelkader El Mahdaouy, Kabil Essefar, Nabil El Mamoun, Ismail Berrada, and Ahmed Khoumsi. 2021. Bert-based multi-task model for country and province level msa and dialectal arabic identification. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 271– 275.
  18. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/salam-2022-dialect-sentiment-identification-nuanced-arabic/