Elamur Research⌕
☰
home / papers / 2021 / BERT-based Multi-Task Model for Country and Province Level Modern Standard Arabic and Dialectal Arabic Identification

BERT-based Multi-Task Model for Country and Province Level Modern Standard Arabic and Dialectal Arabic Identification

Summary

The paper describes a system entered in the second NADI shared task (NADI 2021). The task is to guess where an Arabic tweet comes from, at the country level and at the province level, with separate tracks for tweets in the formal standard variety (MSA) and tweets in local dialect (DA). The authors fine-tune MARBERT, a BERT variant pretrained on about a billion Arabic tweets, as one encoder shared by two tasks. Each task gets its own attention layer that pools token embeddings. The pooled vector is concatenated with the [CLS] embedding and passed to a small task-specific classifier. Training minimizes the sum of the country and province cross-entropy losses. The data has 31,000 tweets labeled with 21 countries and 100 provinces. Compared with a single-task model built the same way, the multi-task version scores higher on every subtask in both macro-F1 and accuracy. The gains are small, and absolute scores stay low, especially for provinces. The work shows that joint training on nested geographic labels helps fine-grained dialect identification.

Key points

  • NADI 2021 has four subtasks: country-level MSA (1.1), country-level DA (1.2), province-level MSA (2.1) and province-level DA (2.2). The data split is 21,000 tweets for training and 10,000 divided equally between dev and test. Country labels are heavily imbalanced, with Egypt, Iraq and Saudi Arabia dominating.
  • The architecture has three parts:
    • a shared MARBERT encoder;
    • two additive-style attention layers, one per task, over the contextual token embeddings;
    • two classifiers with one hidden layer each, whose input is the attention vector concatenated with [CLS].
  • Training uses Adam with a learning rate of 1e-5, batch size 16 and 5 epochs. Preprocessing follows MARBERT's own pipeline: diacritics are stripped and user mentions become a USER token.
  • Test macro-F1, single-task vs. multi-task:
    • country MSA: 20.90 vs. 21.47;
    • country DA: 29.07 vs. 30.64;
    • province MSA: 4.72 vs. 5.35;
    • province DA: 5.30 vs. 7.32.
  • In both setups, adding the attention pooling beat classifying from [CLS] alone. The errors concentrate in three places:
    • countries with little training data, such as Djibouti and Bahrain;
    • neighbouring Gulf countries;
    • MSA in general, which carries little regional signal.

Interesting ideas

  • I1. Each task reads the shared encoder through its own learned attention pooling, and that view is combined with a global summary vector, so tasks can share a backbone yet still focus on different tokens. — For Mova (tree): gparse could share one transformer between UPOS/morphology and arc scoring, giving each head its own attention pooling. This keeps the shared encoder and adds cheap, inspectable per-task weights that show which tokens each decision relied on.
  • I2. Coarse and fine labels of one hierarchy (country, province) are learned jointly with summed losses rather than in a sequential pipeline, so the coarse task regularizes the sparse fine one. — For Mova (tokens): coarse UPOS and fine XPOS/feature bundles could be trained jointly, letting the frequent coarse tag support rare fine-grained morphology labels instead of chaining two separate taggers.
  • I3. The single-task baseline is given the identical attention head, so any improvement can be credited to multi-task sharing rather than to extra architecture. — For Mova (learn): ablation runs should use control models with the same capacity, isolating whether silver data or teacher tiers actually cause a gain.
  • I4. Confusion matrices are read against external structure (training frequency, geographic closeness) to explain where the model fails, not just how often. — For Mova (frames): VerbNet role or class errors could be grouped by class frequency and by closeness in the class hierarchy, separating data-scarcity failures from genuinely near-synonymous distinctions.
  • I5. Input cleaning copies exactly the normalization the pretrained encoder saw, which avoids a mismatch between pretraining and fine-tuning text. — For Mova (tokens): normalization of quotes, mentions and diacritics should be aligned with whatever pretrained encoder gparse consumes, so the tokenizer does not silently shift the input distribution.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2106.12495 · arXiv 2106.12495

@misc{mekki2021bert,
  title = {{BERT-based Multi-Task Model for Country and Province Level Modern Standard Arabic and Dialectal Arabic Identification}},
  author = {Mekki, Abdellah El and Mahdaouy, Abdelkader El and Essefar, Kabil and Mamoun, Nabil El and Berrada, Ismail and Khoumsi, Ahmed},
  year = {2021},
  doi = {10.48550/arxiv.2106.12495},
  eprint = {2106.12495},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.12495}
}

References

18 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ## Conclusion In this paper, we introduced our submitted system to the second NADI shared task. We proposed an MTL model for joint country-level and province-Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020a. ARBERT & MARBERT: Deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗ our card
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020b. NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP 2020), Barcelona, Spain. find
  3. Nizar Habash, Mona Diab, and Owen Rambow. 2012. Conventional orthography for dialectal Arabic. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 711–718, Istanbul, Turkey. European Languages Resources Association (ELRA). find our card
  4. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. NADI 2021: The Second Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP 2021). find
  5. Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. find
  6. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, and Lyle Ungar. 2020c. Toward micro-dialect identification in diaglossic and code-switched environments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5855–5876, Online. Association for Computational Linguistics. find
  7. Kathrein Abu Kwaik and Motaz Saad. 2019. ArbDialectID at MADAR shared task 1: Language modelling and ensemble learning for fine grained Arabic dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 254–258, Florence, Italy. Association for Computational Linguistics. find our card
  8. Imad A. Al-Sughaiyer and Ibrahim A. Al-Kharashi. 2004. Arabic morphological analysis techniques: A comprehensive survey. Journal of the American Society for Information Science and Technology, 55(3):189–213. find
  9. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. ↗ our card
  10. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207. find our card
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  12. Abdellah El Mekki, Ahmed Alami, Hamza Alami, Ahmed Khoumsi, and Ismail Berrada. 2020. Weighted combination of BERT and n-GRAM features for nuanced Arabic dialect identification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 268–274, Barcelona, Spain (Online). Association for Computational Linguistics. find
  13. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015. ↗
  14. Karima Meftouh, Karima Abidi, Salima Harrat, and Kamel Smaili. 2019. The SMarT classifier for Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 259–263, Florence, Italy. Association for Computational Linguistics. find
  15. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
  16. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein Al-Natsheh. 2020. Multi-dialect Arabic BERT for country-level dialect identification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 111–118, Barcelona, Spain (Online). Association for Computational Linguistics. find
  17. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480–1489, San Diego, California. Association for Computational Linguistics. find our card
  18. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mekki-2021-bert-based-multi-task-model-country/