Elamur Research⌕
☰
home / papers / 2025 / Evaluating Calibration of Arabic Pre-trained Language Models on Dialectal Text

Evaluating Calibration of Arabic Pre-trained Language Models on Dialectal Text

Paper · 2025 · Ali Al-Laith and Rachida Kebdani

Summary

The paper looks at a neglected side of Arabic dialect identification: whether the confidence scores of pre-trained language models can be trusted, not just whether their predictions are correct. The authors take eight publicly available fine-tuned dialect-identification PLMs, including CAMeLBERT-MSA DID MADAR Twitter-5, CAMeLBERT-Mix DID NADI and ADI-NADI-2023. They test them on two kinds of data. The first is over 1 million social-media and corpus texts, labelled automatically and kept only where all eight models give the same dialect. The second is the human-annotated NADI-2023 set. Calibration is measured with Expected Calibration Error (ECE), both per model and per dialect. Calibration quality differs widely: some models stay close to their stated confidence, while others are badly overconfident. It also shifts with annotation type and with region. Palestinian and Sudanese Arabic are flagged as especially problematic. The study argues that model selection for sensitive uses such as sentiment analysis or social media monitoring should consider calibration alongside accuracy.

Key points

  • Automatic portion of the data:
    • over 360,000 sentences from the Arabic Dialect Identification dataset, which merges arabic_pos_dialect, IADD, QADI and MADAR;
    • more than 500,000 tweets from AraSenCorpus;
    • over 200,000 tweets from a 5.5-million-tweet emotion/symptom corpus;
    • cleaning removed URLs, hashtags, mentions and duplicates.
  • Manual portion: the NADI 2023 train and dev sets, 15,400 samples covering 14 dialects, with 1,100 per dialect.
  • Predictions come from a softmax over logits, and the top probability serves as confidence. Silver labels are kept only when all eight models agree.
  • Reported ECE values in one evaluation:
    • the worst model reaches 0.45;
    • Models 2 and 5 score 0.21 and 0.31;
    • Model 3 scores 0.08;
    • Models 6, 7 and 8 score 0.02, 0.06 and 0.02.
    • Across both datasets, Models 1, 6 and 8 look most reliable, while Models 4 and 5 degrade most on manual labels.
  • The authors name these limitations:
    • ECE is the only metric; they suggest Brier Score and MCE as additions;
    • the automatic sample is not balanced across dialects;
    • code-switching and sociolinguistic factors are not modelled.

Interesting ideas

  • I1. Building a large silver set by keeping only items on which every one of several independent models agrees trades coverage for label confidence without any human effort.
  • I2. Measuring the gap between a model's stated confidence and its actual hit rate (ECE) exposes models that are accurate on average but dangerously overconfident.
  • I3. Breaking calibration down per class (here per dialect) reveals pockets where a model is unreliable even when its overall ECE looks fine.
  • I4. Comparing the same models' calibration on machine-agreed labels versus human gold labels shows that silver data can make models look better calibrated than they are.
  • I5. The authors point out that ECE alone hides behaviour at specific confidence levels and under class imbalance, and recommend adding Brier Score and MCE.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Our multi-witness parser agreement could adopt a strict "all witnesses agree" tier as the highest-trust silver layer, kept separate from majority-vote data.
  • I2 (tree): Scoring gparse and the perceptron parser with ECE on attachment/label confidences would tell us when an arc score can safely gate downstream frame building.
  • I3 (tree): Reporting calibration per UD relation or per treebank/genre would show which dependency labels need repair rules or a fallback before they reach frames.
  • I4 (learn): Every calibration or accuracy figure computed on silver data should be paired with a human-gold check, serving as a built-in negative control against self-confirming evaluation.
  • I5 (answer): When a reader reports confidence in a test answer, tracking Brier and max-bin error alongside ECE would catch rare but severe overconfident wrong answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Ali Al-Laith and Rachida Kebdani. 2025. Evaluating Calibration of Arabic Pre-trained Language Models on Dialectal Text. Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{allaith2025evaluating,
  title = {{Evaluating Calibration of Arabic Pre-trained Language Models on Dialectal Text}},
  author = {Ali Al-Laith and Rachida Kebdani},
  booktitle = {Proceedings of the 4th Workshop on Arabic Corpus Linguistics (WACL-4)},
  year = {2025},
  url = {https://aclanthology.org/2025.wacl-1.8/}
}

References

35 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amine Abdaoui, Mohamed Berrimi, Mourad Oussalah, and Abdelouahab Moussaoui. 2021. Dziribert: a pre-trained language model for the algerian dialect. arXiv preprint arXiv:2109.12346. ↗
  2. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of arabic. In LREC, pages 1240–1245.
  3. Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2020. Arabic dialect identification in the wild. arXiv preprint arXiv:2005.06557. ↗
  4. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, El Moatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023a. Nadi 2023: The fourth nuanced arabic dialect identification shared task. arXiv preprint arXiv:2310.16117. ↗
  5. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find
  6. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, El Moatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023b. Nadi 2023: The fourth nuanced arabic dialect identification shared task. arXiv preprint arXiv:2310.16117. ↗
  7. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The madar shared task on arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199– 207.
  8. Muhammad Abdul-Mageed, Amr Keleg, AbdelRahim Elmadany, Chiyu Zhang, Injy Hamed, Walid Magdy, Houda Bouamor, and Nizar Habash. 2024. Nadi 2024: The fifth nuanced arabic dialect identification shared task. arXiv preprint arXiv:2407.04910. ↗
  9. Shrey Desai and Greg Durrett. 2020a. Calibration of pre-trained transformers. arXiv preprint arXiv:2003.07892. ↗
  10. Shrey Desai and Greg Durrett. 2020b. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295–302, Online. Association for Computational Linguistics. find
  11. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. Nadi 2022: The third nuanced arabic dialect identification shared task. arXiv preprint arXiv:2210.09582. ↗
  12. Mukhtar Elgezouli, Khalid N Elmadani, and Muhammed Saeed. 2021. Sudabert: A pre-trained encoder representation for sudanese arabic dialect. In 2020 International Conference on Computer, Control, Electrical, and Electronics Engineering (ICCCEEE), pages 1–4. IEEE. find
  13. Murtadha Ahmed, Saghir Alfasly, Bo Wen, Jamaal Qasem, Mohammed Ahmed, and Yunfeng Liu. 2024. Alclam: Arabic dialectal language model. arXiv preprint arXiv:2407.13097. ↗
  14. Ali Al-Laith and Mamdouh Alenezi. 2021. Monitoring people’s emotions and symptoms from arabic tweets during the covid-19 pandemic. Information, 12(2):86. ↗
  15. Hatem Haddad, Ahmed Cheikh Rouhou, Abir Messaoudi, Abir Korched, Chayma Fourati, Amel Sellami, Moez Ben HajHmida, and Faten Ghriss. 2023. Tunbert: pretraining bert for tunisian dialect understanding. SN Computer Science, 4(2):194. ↗
  16. Ali Al-Laith, Muhammad Shahbaz, Hind F Alaskar, and Asim Rehmat. 2021. Arasencorpus: A semi-supervised approach for sentiment annotation of a large arabic text corpus. Applied Sciences, 11(5):2434. ↗
  17. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678. ↗
  18. Reem AlYami and Rabah Al-Zaidy. 2022. Weakly and semi-supervised learning for Arabic text classification using monodialectal language models. In Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), pages 260–272, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  19. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962–977. ↗
  20. Amr Keleg and Walid Magdy. 2023. Arabic dialect identification under scrutiny: Limitations of single-label classification. arXiv preprint arXiv:2310.13661. ↗
  21. Jihad Zahir. 2022. Iadd: An integrated arabic dialect identification dataset. Data in Brief, 40:107777. ↗
  22. Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Knowing more about questions can help: Improving calibration in question answering. arXiv preprint arXiv:2106.01494. ↗
  23. Salam Khalifa, Nizar Habash, Dana Abdulrahim, and Sara Hassan. 2016. A large scale corpus of gulf arabic. arXiv preprint arXiv:1609.02960. ↗
  24. Adrien LeCoz, Stéphane Herbin, and Faouzi Adjed. 2024. Confidence calibration of classifiers with many classes. Preprint, arXiv:2411.02988. ↗
  25. Yingjie Li and Cornelia Caragea. 2023. Distilling calibrated knowledge for stance detection. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6316–6329.
  26. Otman Moussaoui and Yacine El Younnoussi. 2023. Pre-training two bert-like models for moroccan dialect: Morroberta and morrbert. In MENDEL, volume 29, pages 55–61. ↗
  27. Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29. find
  28. Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. 2019. Measuring calibration in deep learning. In CVPR workshops, volume 2. find
  29. Faisal Qarah. 2024a. Egybert: A large language model pretrained on egyptian dialect corpora. arXiv preprint arXiv:2408.03524. ↗
  30. Faisal Qarah. 2024b. Saudibert: A large language model pretrained on saudi dialect corpora. arXiv preprint arXiv:2405.06239. ↗
  31. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media. arXiv preprint arXiv:2007.13184. ↗
  32. Guokan Shang, Hadi Abdine, Yousef Khoubrane, Amr Mohamed, Yassine Abbahaddou, Sofiane Ennadir, Imane Momayiz, Xuguang Ren, Eric Moulines, Preslav Nakov, et al. 2024. Atlas-chat: Adapting large language models for low-resource moroccan arabic dialect. arXiv preprint arXiv:2409.17912. ↗
  33. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein T Al-Natsheh. 2020. Multi-dialect arabic bert for country-level dialect identification. arXiv preprint arXiv:2007.05612. ↗
  34. Yahan Yang, Soham Dan, Dan Roth, and Insup Lee. 2023. Understanding calibration for multilingual question answering models. arXiv preprint arXiv:2311.08669. ↗
  35. Wajdi Zaghouani and Anis Charfi. 2018. Arap-tweet: A large multi-dialect twitter corpus for gender, age and language variety identification. arXiv preprint arXiv:1808.07674. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/laith-2025-evaluating-calibration-arabic-pre-trained/