Elamur Research⌕
☰
home / papers / 2025 / mucAI at BAREC Shared Task 2025: Towards Uncertainty Aware Arabic Readability Assessment

mucAI at BAREC Shared Task 2025: Towards Uncertainty Aware Arabic Readability Assessment

Paper · 2025 · Ahmed Abdou

Summary

The paper tackles the BAREC 2025 Shared Task, which asks systems to assign Arabic text to detailed readability levels. Each sentence must be placed on one of 19 ordered difficulty levels, and scoring uses Quadratic Weighted Kappa (QWK). Under QWK, predictions far from the gold level cost far more than near misses. The author fine-tunes AraBERT-v2 on Farasa-segmented text with several objectives: Cross-Entropy, EMD, a Focal loss added for the skewed label distribution, and Regression. On top of these models sits a post-processing step that works with any base model. Conformal prediction first builds a label set that should contain the true level with a chosen probability. The model's probabilities are then renormalized over that set, and the rounded expected level becomes the final prediction. This decoding lowers exact accuracy but pulls distant mistakes closer to the gold level. The result is a QWK gain of 1–3 points for every base model. In the strict track the system reaches 84.9 QWK on test and 85.7 on blind test for sentences, and 73.3 for documents. The compact label sets also give human reviewers a short list of plausible levels to check.

Key points

  • The BAREC corpus has over 69k sentences annotated with 19 levels. The splits are 54.8k train, 7.3k dev, 7.3k test and 3.4k blind. In the strict track, only BAREC data may be used for training.
  • The official dev set is divided into two stratified halves. Dev-cal sets the conformal thresholds, and dev-tune is used for choosing hyperparameters. The paper compares three nonconformity scores: naïve, APS and RAPS. The best setting is Focal loss with APS at α=0.1.
  • On the test set, Focal+CP scores 84.4 QWK, a gain of 2.6 over Focal alone. Probability-averaging ensembles reach 84.9. A non-deployable Oracle, which picks the gold label whenever it lies inside the set, reaches 95.3 QWK.
  • Empirical coverage is 94.88% against a 90% target, with sets averaging about 5 of the 19 levels. Coverage misses vary by domain: 4.3% for Arts & Humanities, 6.1% for STEM and 7.1% for Social Sciences.
  • On dev-tune, CP turned 362 correct predictions into errors, and 86.7% of these were only one level off. It also improved 397 wrong predictions, some by 2–4 levels. The authors say errors at levels 15–19 largely remain.

Interesting ideas

  • I1. Instead of taking the argmax, average the model's probabilities over a calibrated set of plausible labels; this suits any ordered output where far-off errors cost more than near ones.
  • I2. Conformal sets can stand in for one-best outputs, giving a short candidate list with a coverage guarantee that downstream steps or humans can choose from.
  • I3. Splitting the held-out data into a calibration half and a tuning half keeps threshold fitting apart from model selection.
  • I4. An "oracle within the set" upper bound shows how much could be gained if a perfect chooser picked from the candidates.
  • I5. Breaking down coverage failures by domain and text class exposes where calibration is weak, and the paper suggests per-group (Mondrian) calibration as a fix.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Numeric answers to maths word problems could be decoded as the expected value over a calibrated candidate set rather than the single top guess.
  • I2 (tree): gparse or the perceptron parser could output a calibrated set of head/label candidates per token, so later frame building can recover when the top arc is wrong.
  • I3 (learn): When silver-trained models are calibrated, a separate calibration slice would keep coverage claims honest and independent of hyperparameter tuning.
  • I4 (learn): Measuring an oracle over witness-parser candidate sets would show whether a better selection rule or repair rule is worth building.
  • I5 (tokens): Calibration misses of the tagger and lemmatizer could be reported per genre (school texts, fables, news) to decide where separate thresholds are needed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksEducation · 203 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ahmed Abdou. 2025. mucAI at BAREC Shared Task 2025: Towards Uncertainty Aware Arabic Readability Assessment. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.44

@inproceedings{abdou2025mucai,
  title = {{mucAI at BAREC Shared Task 2025: Towards Uncertainty Aware Arabic Readability Assessment}},
  author = {Ahmed Abdou},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.44},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.44}
}

References

19 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16.
  2. Bashar Alhafni, Reem Hazim, Juan David Pineros Liberato, Muhamed Al Khalil, and Nizar Habash. 2024. The SAMER Arabic text simplification corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16079–16093, Torino, Italia. ELRA and ICCL.
  3. Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I Jordan. 2020. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14193. ↗
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  5. Yassine Benajiba and Paolo Rosso. 2008. Arabic named entity recognition using conditional random fields. In Proc. of Workshop on HLT & NLP within the Arabic World, LREC, volume 8, pages 143–153. find
  6. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213. ↗
  7. Kevyn Collins-Thompson and Jamie Callan. 2004. Information retrieval for language tutoring: An overview of the reap project. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 544–545. find
  8. Khalid Elmadani, Nizar Habash, and Hanada Taha. 2025a. A large and balanced corpus for fine-grained Arabic readability assessment. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16376–16400, Vienna, Austria. Association for Computational Linguistics.
  9. Khalid N. Elmadani, Bashar Alhafni, Hanada Taha, and Nizar Habash. 2025b. BAREC shared task 2025 on Arabic readability assessment. In Proceedings of the Third Arabic Natural Language Processing Conference, Suzhou, China. Association for Computational Linguistics.
  10. Nizar Habash, Hanada Taha-Thomure, Khalid Elmadani, Zeina Zeino, and Abdallah Abushmaes. 2025. Guidelines for fine-grained sentence-level Arabic readability annotation. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages 359–376, Vienna, Austria. Association for Computational Linguistics.
  11. Le Hou, Chen-Ping Yu, and Dimitris Samaras. 2017. Squared earth movers distance loss for training deep neural networks on ordered-classes. In NIPS workshop, volume 5. find
  12. Juan Liberato, Bashar Alhafni, Muhamed Khalil, and Nizar Habash. 2024. Strategies for Arabic readability modeling. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 55–66, Bangkok, Thailand. Association for Computational Linguistics.
  13. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988. ↗
  14. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. Camel tools: An open source python toolkit for arabic natural language processing. In Proceedings of the twelfth language resources and evaluation conference, pages 7022–7032.
  15. Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. 2002. Inductive confidence machines for regression. In European conference on machine learning, pages 345–356. Springer. find
  16. Yaniv Romano, Matteo Sesia, and Emmanuel Candes. 2020. Classification with valid and adaptive coverage. Advances in neural information processing systems, 33:3581–3591. ↗
  17. Hind Saddiki, Nizar Habash, Violetta Cavalli-Sforza, and Muhamed Al-Khalil. 2018. Feature optimization for predicting readability of arabic l1 and l2. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, pages 20–29. ↗
  18. Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. 2005. Algorithmic learning in a random world. Springer. ↗
  19. Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2016. Text readability assessment for second language learners. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 12–22, San Diego, CA. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abdou-2025-mucai-barec-shared-task-2025/