Elamur Research⌕
☰
home / papers / 2022 / On The Arabic Dialects' Identification: Overcoming Challenges of Geographical Similarities Between Arabic dialects and Imbalanced Datasets

On The Arabic Dialects' Identification: Overcoming Challenges of Geographical Similarities Between Arabic dialects and Imbalanced Datasets

Paper · 2022 · Salma Jamal, Aly M. Kassem, Omar Mohamed et al.

Summary

The paper describes a system for the NADI 2022 shared task, subtask 1: country-level Arabic dialect identification on tweets. It targets two difficulties. First, dialects of neighbouring countries share much vocabulary, and some of the shared words mean different things in different dialects. Second, the training data is heavily skewed toward a few countries.

The pipeline has three parts:

  • Cleaning: removes non-Arabic characters, digits, diacritics, character elongation and emojis.
  • Frequency filtering: drops every word whose corpus count exceeds a tuned threshold, on the assumption that very frequent words are shared across dialects and confuse the classifier.
  • Classifier: MARBERT-v2 is fine-tuned with a weighted sum of VS loss and weighted cross-entropy.

On the development set, filtering combined with the hybrid loss gave the best macro F1. The system placed third, with 35.68% macro F1 on test A, 17.192% on test B and 26.44% averaged. The confusion matrix shows that errors cluster among geographically close countries, such as the Gulf states. The work shows that cheap data-side and loss-side fixes can help on imbalanced, fine-grained label sets.

Key points

  • Data (NADI-2022 subtask 1):
    • 18 dialects, about 20k training tweets and 4,871 development tweets.
    • Test A has 4,758 tweets; test B has 1,474 tweets with an unknown number of dialects.
    • Egypt has 4,283 tweets, while Sudan, Qatar and Bahrain have only 215 each.
  • Development-set macro F1 (cleaning only):
    • Weighted CE: 28.315
    • VS loss: 34.207
    • Hybrid loss: 34.274
  • Development-set macro F1 (with frequent-term removal):
    • VS loss: 34.461
    • Hybrid loss: 35.8 (best result)
  • Hybrid loss: fixed weights of 0.7 for VS loss and 0.3 for weighted cross-entropy. Both losses come from separate predictions, and gradients from both are backpropagated together.
  • Model: MARBERT-v2, a BERT-base architecture with 163M parameters and no NSP, pretrained on about 1B multidialectal tweets plus other Arabic corpora, with sequence length 512.
  • Error analysis and limitations:
    • The KSA dialect is mostly confused with UAE, Kuwaiti and Omani.
    • The authors note that the dataset size limits generalisation.
    • They also note that filtering by frequency alone is crude.

Interesting ideas

  • I1. The cutoff for "too common to be informative" is treated as a tunable hyperparameter and swept on dev data rather than fixed by a stop-word list.
  • I2. Two imbalance-aware losses (VS and weighted CE) are combined with fixed weights, on the argument that they make different errors and so act like a cheap in-model ensemble.
  • I3. The ablation is laid out as a small grid of preprocessing choice × loss choice, which isolates each component's contribution.
  • I4. The confusion matrix is checked against an external structure (geography) to confirm that errors are systematic rather than random.
  • I5. The paper shows that one surface word can carry different senses depending on the variety of the speaker.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): when converting or comparing UD dialects across treebanks, a swept frequency cutoff could separate lemmas and constructions shared by all varieties from those marking one treebank's annotation style.
  • I2 (tree): gparse could train deprel classification with a similar blended loss, so rare relations (e.g. vocative, dislocated) are not drowned out by nsubj/obj.
  • I3 (learn): evaluations of silver-data filters and training objectives could use the same factorial layout, so each gain is attributed to a specific change.
  • I4 (learn): parser and tagger confusions could be checked against linguistic groupings (e.g. obl vs obj, amod vs compound) to tell structured ambiguity from noise and target repair rules accordingly.
  • I5 (world): sense selection from WordNet could take a register or variety cue (e.g. British vs American usage) as an explicit, explainable input instead of assuming one global default sense.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nile UniversityGiza · Egypt14 works · 158 citations🎓Helwan UniversityCairo · Egypt8 works · 36 citations🎓University of WindsorWindsor · Canada2 works · 258 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Salma Jamal, Aly M. Kassem, Omar Mohamed and Ali Ashraf. 2022. On The Arabic Dialects' Identification: Overcoming Challenges of Geographical Similarities Between Arabic dialects and Imbalanced Datasets. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.49

@inproceedings{jamal2022arabic,
  title = {{On The Arabic Dialects' Identification: Overcoming Challenges of Geographical Similarities Between Arabic dialects and Imbalanced Datasets}},
  author = {Salma Jamal, Aly M. Kassem, Omar Mohamed and Ali Ashraf},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.49},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.49}
}

References

17 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2021. Qadi: Arabic dialect identification in the wild. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 1–10.
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021a. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  4. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021b. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  5. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Task. In Proceedings of the Seven Arabic Natural Language Processing Workshop (WANLP 2022).
  6. Badr AlKhamissi, Mohamed Gabr, Muhammad El-Nokrashy, and Khaled Essam. 2021. Adapting marbert for improved arabic dialect identification: Submission to the nadi 2021 shared task. arXiv preprint arXiv:2103.01065. ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  8. Ibrahim Abu El-Khair. 2016. 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033. ↗
  9. Abdellah El Mekki, Ahmed Alami, Hamza Alami, Ahmed Khoumsi, and Ismail Berrada. 2020. Weighted combination of bert and n-gram features for nuanced arabic dialect identification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 268–274.
  10. Ali Mostafa, Omar Mohamed, and Ali Ashraf. 2022. Gof at arabic hate speech 2022: Breaking the loss function convention for data-imbalanced arabic offensive text detection. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 167–175, Marseille, France. European Language Resources Association.
  11. El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Tariq Alhindi. 2020. Machine generation and detection of Arabic manipulated and fake news. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 69–84, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  12. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache. find
  13. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein T Al-Natsheh. 2020. Multi-dialect arabic bert for country-level dialect identification. arXiv preprint arXiv:2007.05612. ↗
  14. Omar Zaidan and Chris Callison-Burch. 2011. The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41.
  15. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy. Association for Computational Linguistics. ↗
  16. Abdellah El Mekki, Abdelkader El Mahdaouy, Kabil Essefar, Nabil El Mamoun, Ismail Berrada, and Ahmed Khoumsi. 2021. Bert-based multi-task model for country and province level msa and dialectal arabic identification. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 271– 275.
  17. Ganesh Ramachandra Kini, Orestis Paraskevas, Samet Oymak, and Christos Thrampoulidis. 2021. Label-imbalanced and group-sensitive classification under overparameterization. In Advances in Neural Information Processing Systems, volume 34, pages 18970– 18983. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jamal-2022-arabic-dialects-identification-overcoming-challenges/