Elamur Research⌕
☰
home / papers / 2024 / AlexUNLP-STM at NADI 2024 shared task: Quantifying the Arabic Dialect Spectrum with Contrastive Learning, Weighted Sampling, and BERT-based Regression Ensemble

AlexUNLP-STM at NADI 2024 shared task: Quantifying the Arabic Dialect Spectrum with Contrastive Learning, Weighted Sampling, and BERT-based Regression Ensemble

Paper · 2024 · Abdelrahman Sakr, Marwan Torki and Nagwa El-Makky

Summary

The paper describes a system for Subtask 2 of NADI 2024, which asks how dialectal an Arabic sentence is. The answer is a continuous Arabic Level of Dialectness (ALDi) score from 0 (pure Modern Standard Arabic) to 1 (strongly dialectal), not a binary MSA-vs-dialect label. The authors trained on AOC-ALDi, which is built from reader comments in Egyptian, Levantine and Gulf newspapers. Over half of its labels are exactly 0, and that skew pushes models to overfit to MSA. Their approach has four parts:

  • fine-tune Arabic BERT encoders (AraBERTv2-Twitter, MARBERTv2) with a small multilayer regression head;
  • counter the label skew with an inverse-frequency weighted random sampler;
  • add an NT-Xent contrastive term on the pooled representation alongside the RMSE objective;
  • combine checkpoints from different training setups by taking the median of their predictions.

The final ensemble reached a test RMSE of 0.1406, second on the leaderboard. The work shows that cheap fixes for imbalance and representation shaping matter as much as encoder choice for graded linguistic judgments.

Key points

  • Training data: 102,886 samples with 68 distinct label values. 53,844 of them (about 52.3%) are labelled 0. The authors split it 85/15 into their own train and dev parts. The official dev set has only 107 items and the blind test set has 857.
  • Architecture and training:
    • The regressor runs 768→512→256→1 with ReLU and dropout 0.2.
    • Training used AdamW with learning rate 5e-6.
    • Bounding outputs with a sigmoid made results worse, as did stripping non-Arabic characters and links (RMNALL).
  • Encoder choice: AraBERTv2-Twitter-Base beat the Large version and MARBERTv2 once weighted sampling was added. The authors attribute this to the larger model overfitting the oversampled minority items.
  • Contrastive loss: NT-Xent was tried with temperatures 0.3, 0.5 and 1. Pairing it with weighted sampling gave the best single models. A variant that also dropped about 43,000 MSA samples (DS) was tested as well.
  • Ensembling: Five checkpoints were combined. The median beat both averaging and voting, giving 0.1406 test RMSE. The authors note a dev/test gap that they blame on the tiny official dev set.

Interesting ideas

  • I1. Sampling training items with probability inversely proportional to their label's frequency rebalances a skewed graded target without discarding data, but it should be paired with a smaller model so the oversampled rare items are not memorized.
  • I2. Feeding the same pooled vector into both the task head and a label-aware contrastive loss shapes the embedding space so that items with similar targets cluster, which helps a regressor on imbalanced data.
  • I3. Taking the median of several diverse checkpoints is a simple aggregation that resists a single badly-off model better than averaging.
  • I4. "Noise" such as Latin characters or links turned out to carry annotation signal, so removing it hurt; cleaning steps should be ablated rather than assumed harmless.
  • I5. Treating a linguistic property as a continuum rather than a binary class yields finer-grained and more honest judgments.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when training on silver data where common constructions dominate, inverse-frequency sampling over rare relation or construction types could lift recall on the tail, with a check that capacity is not overfitting the repeated examples.
  • I2 (tree): gparse could add an auxiliary contrastive term on its encoder states so that tokens sharing a dependency relation sit close together, possibly sharpening rare-label attachment decisions.
  • I3 (learn): when several witness parsers or teacher tiers give numeric scores (confidence, attachment probability), a median rather than a mean would keep one outlier witness from swaying the silver label.
  • I4 (tokens): the tokenizer should keep URLs, code-switched words and odd spellings as typed tokens rather than stripping them, since they may cue register or informality downstream.
  • I5 (frames): speech-act and irony detection in crate prag could output graded strengths (e.g., how ironic or how formal) instead of yes/no tags, keeping each score explainable by the cues that raised it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Alexandria UniversityAlexandria · Egypt39 works · 287 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Abdelrahman Sakr, Marwan Torki and Nagwa El-Makky. 2024. AlexUNLP-STM at NADI 2024 shared task: Quantifying the Arabic Dialect Spectrum with Contrastive Learning, Weighted Sampling, and BERT-based Regression Ensemble. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.81

@inproceedings{sakr2024alexunlp,
  title = {{AlexUNLP-STM at NADI 2024 shared task: Quantifying the Arabic Dialect Spectrum with Contrastive Learning, Weighted Sampling, and BERT-based Regression Ensemble}},
  author = {Abdelrahman Sakr, Marwan Torki and Nagwa El-Makky},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.81},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.81}
}

References

28 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2021. QADI: Arabic dialect identification in the wild. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 1–10, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021a. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021b. Arbert marbert: Deep bidirectional transformers for arabic. Preprint, arXiv:2101.01785. ↗
  4. Muhammad Abdul-Mageed, Amr Keleg, AbdelRahim Elmadany, Chiyu Zhang, Injy Hamed, Walid Magdy, Houda Bouamor, and Nizar Habash. 2024. NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024).
  5. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  7. Shiv Ram Dubey, Satish Kumar Singh, and Bidyut Baran Chaudhuri. 2022. Activation functions in deep learning: A comprehensive survey and benchmark. Preprint, arXiv:2109.14545. ↗
  8. Heba Elfardy and Mona Diab. 2013. Sentence level dialect identification in Arabic. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 456–461, Sofia, Bulgaria. Association for Computational Linguistics.
  9. Nizar Y Habash. 2010. Introduction to Arabic natural language processing. Morgan & Claypool Publishers. ↗
  10. Amr Keleg, Sharon Goldwater, and Walid Magdy. 2023. ALDi: Quantifying the Arabic level of dialectness of text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10597–10611, Singapore. Association for Computational Linguistics.
  11. Abien Fred Agarap. 2018. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375. ↗
  12. Amr Keleg and Walid Magdy. 2023. Arabic dialect identification under scrutiny: Limitations of single-label classification. In Proceedings of ArabicNLP 2023, pages 385–398, Singapore (Hybrid). Association for Computational Linguistics.
  13. Maha J. Althobaiti. 2020. Automatic arabic dialect identification systems for written texts: A survey. CoRR, abs/2009.12622. ↗
  14. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101. ↗
  15. Maha J. Althobaiti. 2022. Creation of annotated country-level dialectal arabic resources: An unsupervised approach. Natural Language Engineering, 28(5):607–648. ↗
  16. Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey-Villamizar, Mona Diab, and Thamar Solorio. 2016. Overview for the second shared task on language identification in code-switched data. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 40–49, Austin, Texas. Association for Computational Linguistics.
  17. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  18. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR. ↗
  19. Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie. 2022. When does contrastive visual representation learning work? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14755–14764. find
  20. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc. ↗
  21. Silvia L Pintea, Yancong Lin, Jouke Dijkstra, and Jan C van Gemert. 2023. A step towards understanding why classification helps regression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19972–19981. find
  22. A. Sakr and M. Torki. 2023. Arapunc: Arabic punctuation restoration using transformers. In 2023 20th ACS/IEEE International Conference on Computer Systems and Applications (AICCSA), pages 1–6, Los Alamitos, CA, USA. IEEE Computer Society. find
  23. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332– 1344, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  24. Ahmad Shapiro, Ayman Khalafallah, and Marwan Torki. 2022. AlexU-AIC at Arabic hate speech 2022: Contrast to classify. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 200–208, Marseille, France. European Language Resources Association. ↗
  25. Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, and Pascale Fung. 2014. Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 62–72, Doha, Qatar. Association for Computational Linguistics.
  26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing. Preprint, arXiv:1910.03771. ↗
  27. Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41, Portland, Oregon, USA. Association for Computational Linguistics.
  28. Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014. A report on the DSL shared task 2014. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, pages 58–67, Dublin, Ireland. Association for Computational Linguistics and Dublin City University. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sakr-2024-alexunlp-stm-nadi-2024-shared/