Elamur Research⌕
☰
home / papers / 2024 / ELYADATA at NADI 2024 shared task: Arabic Dialect Identification with Similarity-Induced Mono-to-Multi Label Transformation.

ELYADATA at NADI 2024 shared task: Arabic Dialect Identification with Similarity-Induced Mono-to-Multi Label Transformation.

Paper · 2024 · Amira Karoui, Farah Gharbi, Rami Kammoun et al.

Summary

NADI 2024 introduced multi-label country-level Arabic dialect identification (MLDID), in which one tweet may be valid in several countries' dialects. The training data stayed single-label: the NADI-2023 set of 18 dialects with 1,000 tweets each. The ELYADATA team closed this gap with SIMMT. For each dialect they built a vocabulary, measured word overlap between every training sentence and each dialect's vocabulary, and added every dialect whose overlap passed a threshold as an extra label. They fine-tuned three Arabic BERT models (MarBERT, ArBERT, MarBERTv2) on the relabeled data and combined them with a Max Probabilities Ensemble that used a very low decision cutoff. Binary Relevance and staged fine-tuning on older NADI data were also tested and dropped. The system took first place with a test macro-F1 of 50.57%. The main lesson is that cheap lexical relabeling can turn single-label training data into usable multi-label supervision when no native multi-label corpus exists.

Key points

  • Data:
    • Train: 18,000 tweets, balanced across 18 dialects, average 16 words, after removing hashtags, laughter tokens, single letters and diacritics.
    • Multi-label dev: 120 sentences over 8 countries.
    • Test: 1,000 sentences.
  • Effect of label format: Mono-label fine-tuning scored only about 0.08–0.09 macro-F1 on the multi-label dev set. Preprocessing helped slightly, with MarBERTv2 reaching 0.091 vs 0.086 unprocessed.
  • Ensembling:
    • Max probabilities scored 0.580 and averaging 0.574.
    • Usual cutoffs (0.3–0.5) gave only about 0.34.
    • A cutoff of 0.0002 was best, because the models put more than 0.9 on one class and near zero on the rest.
  • SIMMT tuning:
    • A 60% similarity threshold gave 0.588 and 40% gave 0.6; 30% did worse.
    • A rule turned "almost all dialects" label vectors into "all dialects," on the reasoning that such sentences are MSA-like. This raised the score from 0.608 to 0.6108.
  • Failed alternatives:
    • Binary Relevance with 18 per-dialect classifiers: 0.49 unbalanced, 0.17 balanced.
    • Staged fine-tuning (NADI-2020 → 2021 → 2023): 55% on dev2, below the chosen system.
    • The authors also report label errors in the dev set.

Interesting ideas

  • I1. Single-label training data can be expanded into multi-label targets by adding any class whose lexicon overlaps the sentence above a threshold; this is cheap supervision for tasks where gold labels are artificially exclusive.
  • I2. Decision thresholds should be fitted to how confident the model's outputs actually are; an overconfident softmax hides secondary labels unless the cutoff is set far below 0.5.
  • I3. Taking the per-class maximum across ensemble members, rather than the mean, keeps a label that any one model supports strongly, which favors recall of plausible extra labels.
  • I4. If an item matches nearly every class, it is better treated as shared or neutral than as a noisy mix of specific classes.
  • I5. Checking the evaluation set itself exposed mislabeled examples, which distort reported scores as well as adding training noise.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When silver data assigns one VerbNet class or sense, lexicon overlap could add other admissible classes, so a student is not punished for valid alternatives.
  • I2 (tree): gparse could expose alternative heads or relation labels by tuning a dev-calibrated low cutoff instead of keeping only the argmax arc.
  • I3 (learn): Witness parsers could be merged with a max rule to flag ambiguous attachments for the LLM teacher tier, while agreement-by-average stays the filter for silver gold.
  • I4 (world): A word or frame that fits almost every verb class or domain could be marked "general" in the compiled knowledge rather than given a scattered list of memberships.
  • I5 (learn): Evaluation could include a routine audit of disagreements between gold and confident predictions, to separate model errors from label errors before trusting a metric.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Amira Karoui, Farah Gharbi, Rami Kammoun, Imen Laouirine and Fethi Bougares. 2024. ELYADATA at NADI 2024 shared task: Arabic Dialect Identification with Similarity-Induced Mono-to-Multi Label Transformation. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.85

@inproceedings{karoui2024elyadata,
  title = {{ELYADATA at NADI 2024 shared task: Arabic Dialect Identification with Similarity-Induced Mono-to-Multi Label Transformation.}},
  author = {Amira Karoui, Farah Gharbi, Rami Kammoun, Imen Laouirine and Fethi Bougares},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.85},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.85}
}

References

16 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Amine Ben Abdallah, Ata Kabboudi, Amir Kanoun, and Salah Zaiem. 2023. Leveraging data collection and unsupervised learning for code-switched tunisian arabic automatic speech recognition. Preprint, arXiv:2309.11327. ↗
  2. Amine Abdaoui, Mohamed Berrimi, Mourad Oussalah, and Abdelouahab Moussaoui. 2021. Dziribert: a pre-trained language model for the algerian dialect. CoRR, abs/2109.12346. ↗
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. Arbert & marbert: Deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  4. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The third nuanced Arabic dialect identification shared task. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 85–97, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  5. Asma Aldrees, Azeddine Chikh, and Jawad Berri. 2016. Comparative evaluation of four multi-label classification algorithms in classifying learning objects. Computer Science Information Technology, 6. ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  7. Richard Maclin and David W. Opitz. 2011. Popular ensemble methods: An empirical study. CoRR, abs/1106.0257. ↗
  8. Shervin Malmasi, Eshrag A. Refaee, and Mark Dras. 2015. Arabic dialect identification using a parallel multidialectal corpus. In International Conference of the Pacific Association for Computaitonal Linguistics. find
  9. Abir Messaoudi, Ahmed Cheikhrouhou, Hatem Haddad, Nourchene Ferchichi, Moez BenHajhmida, Abir Korched, Malek Naski, Faten Ghriss, and Amine Kerkeni. 2021. Tunbert: Pretrained contextualized text representation for tunisian dialect. CoRR, abs/2111.13138. ↗
  10. Ammar Mohammed and Rania Kora. 2023. A comprehensive review on ensemble deep learning: Opportunities and challenges. Journal of King Saud University - Computer and Information Sciences, 35(2):757– 774. ↗
  11. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  12. Ridouane Tachicart, Karim Bouzoubaa, Salima Harrat, and Kamel Smaïli. 2022. Arabic Dialects Morphological Analyzers: A Survey, pages 189–203. Springer International Publishing, Cham. find
  13. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein T Al-Natsheh. 2020. Multi-dialect arabic bert for country-level dialect identification. arXiv preprint arXiv:2007.05612. ↗
  14. Mohab El-karef, Movina Moses, Shinnosuke Tanaka, James Barry, and Geeth Mel. 2023. Nlpeople at nadi 2023 shared task: Arabic dialect identification with augmented context and multi-stage tuning. In Proceedings of ArabicNLP 2023, pages 642–646.
  15. Kamel Gaanoun, Abdou Mohamed Naira, Anass Allak, and Imade Benelallam. 2024. Darijabert: a step Wilson L Taylor. 1953. “cloze procedure”: A new tool for measuring readability. Journalism quarterly, 30(4):415–433. find
  16. Yongquan Yang, Haijun Lv, and Ning Chen. 2023. A survey on ensemble learning under the era of deep learning. Artificial Intelligence Review, 56(6):5545– 5589. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/karoui-2024-elyadata-nadi-2024-shared-task/