Elamur Research⌕
☰
home / papers / 2025 / MarsadLab at BAREC Shared Task 2025: Strict-Track Readability Prediction with Specialized AraBERT Models on BAREC

MarsadLab at BAREC Shared Task 2025: Strict-Track Readability Prediction with Specialized AraBERT Models on BAREC

Paper · 2025 · Shimaa Ibrahim, Md. Rafiul Biswas, Mabrouka Bessghaier et al.

Summary

This paper describes a system for the BAREC 2025 Arabic readability shared task. The task is to assign one of 19 ordered difficulty levels to sentences and documents. The team entered only the Strict track, where training may use only the official splits. That restriction means little data, skewed classes and neighbouring labels that are hard to tell apart.

The system trains in two steps. It starts from public AraBERT readability checkpoints that BAREC had already trained with cross-entropy. Next, training continues using only the Strict-track data, now optimizing Weighted Kappa Loss (WKL), a differentiable stand-in for Quadratic Weighted Kappa (QWK), which is the official metric. Sentences use D3Tok-segmented input and documents use Word input. A document receives the level of its hardest sentence.

Sentence-level QWK is 0.842 on the public test and 0.841 on the blind test, against an official baseline of 0.815. Document-level QWK is 0.828 public and 0.790 blind, against a baseline of 0.620. The work shows that matching the training loss to an ordinal metric pays off cheaply when data is constrained.

Key points

  • Task: BAREC 2025 defines 19 ordinal readability levels at sentence and document scope. The authors submitted to the Sentence Strict and Document Strict tracks and used no external data.

  • Model: an AraBERTv2 encoder with a linear head over [CLS]. Training first converges with CE, then switches to WKL with quadratic weights ((i−j)/(K−1))², K=19. The WKL stage uses a lower learning rate and early stopping on dev QWK.

  • Training setup: one NVIDIA T4 GPU, AdamW at 2×10⁻⁵, batch size 16, warmup ratio 0.1, at most 10 epochs. Each result comes from a single model run with no ensembling.

  • Results:

    LevelPublic QWKBlind QWKOfficial baseline (blind)
    Sentence0.8420.8410.815
    Document0.8280.7900.620

    The roughly 0.04 drop from public to blind at document level is attributed to domain shift and to max-level pooling.

  • Error pattern: most mistakes fall between adjacent levels, and large jumps are rare. The authors say MAE, accuracy@±1, multi-seed significance tests and genre/dialect error analysis are still missing.

Interesting ideas

  • I1. Training WKL directly from a raw encoder was unstable, so the authors first converge with plain cross-entropy and only then switch to the metric-shaped loss.
  • I2. Errors are weighted by the squared distance between the true and predicted level, so a near miss costs little and a far miss costs a lot.
  • I3. A document's label is simply the level of its hardest sentence, which matches how the annotation guidelines define document difficulty.
  • I4. The input granularity is chosen per track: morphologically segmented D3Tok for sentences and plain words for documents.
  • I5. The gap between public and blind test scores is read as a sign of domain shift rather than ignored.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when gparse or a tagger is tuned toward a structured metric such as LAS-like or tree-distance objectives, first converge with ordinary CE and then fine-tune on the metric-aligned loss at a lower learning rate.
  • I2 (answer): grade the readers' answers to ordinal questions (sizes, ages, quantities in word problems) by distance from the gold value, so evaluation separates near misses from absurd answers.
  • I3 (tree): score each sentence's parse difficulty and treat the hardest sentence as the document's risk, sending such documents to gparse or to more witness parsers.
  • I4 (tokens): keep both raw-word and morpheme-split views of the text available, and let each downstream task choose the granularity that suits it.
  • I5 (learn): always keep a hidden, differently sourced test split beside the dev set, and treat a falling score on it as an alarm alongside the negative controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northwestern University in QatarDoha · Qatar24 works · 148 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Shimaa Ibrahim, Md. Rafiul Biswas, Mabrouka Bessghaier and Wajdi Zaghouani. 2025. MarsadLab at BAREC Shared Task 2025: Strict-Track Readability Prediction with Specialized AraBERT Models on BAREC. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.39

@inproceedings{ibrahim2025marsadlab,
  title = {{MarsadLab at BAREC Shared Task 2025: Strict-Track Readability Prediction with Specialized AraBERT Models on BAREC}},
  author = {Shimaa Ibrahim, Md. Rafiul Biswas, Mabrouka Bessghaier and Wajdi Zaghouani},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.39},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.39}
}

References

29 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Ibrahim Abu Farha and Walid Magdy. 2021. Benchmarking transformer-based language models for Arabic sentiment and sarcasm detection. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 21–31, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  4. Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.
  5. Abdel Karim Al Tamimi, Manar Jaradat, Nuha Al-Jarrah, and Sahar Ghanem. 2014. Aari: automatic arabic readability index. Int. Arab J. Inf. Technol., 11(4):370–378. find
  6. Bashar Alhafni, Reem Hazim, Juan David Pineros Liberato, Muhamed Al Khalil, and Nizar Habash. 2024. The SAMER Arabic text simplification corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16079–16093, Torino, Italia. ELRA and ICCL.
  7. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  8. Violetta Cavalli-Sforza, Hind Saddiki, and Naoual Nassiri. 2018. Arabic readability research: Current state and future directions. Procedia Computer Science, 142:38–49. Arabic Computational Linguistics. find
  9. Jordi de la Torre, Domenec Puig, and Aida Valls. 2018. Weighted kappa loss function for multi-class classification of ordinal data in deep learning. Pattern Recognition Letters, 105:144–154. Machine Learning and Applications in Artificial Intelligence. ↗
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  11. Mahmoud El-Haj and Paul Rayson. 2016. OSMAN — a novel Arabic readability metric. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 250–255, Portorož, Slovenia. European Language Resources Association (ELRA). find
  12. Khalid Elmadani, Nizar Habash, and Hanada Taha. 2025a. A large and balanced corpus for fine-grained Arabic readability assessment. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16376–16400, Vienna, Austria. Association for Computational Linguistics.
  13. Khalid N. Elmadani, Bashar Alhafni, Hanada Taha, and Nizar Habash. 2025b. BAREC shared task 2025 on Arabic readability assessment. In Proceedings of the Third Arabic Natural Language Processing Conference, Suzhou, China. Association for Computational Linguistics.
  14. Lijun Feng, Michael Elhadad, and Matt Huenerfauth. 2010. Automatic readability assessment for text simplification. In Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–62.
  15. Jonathan Forsyth. 2014. Automatic readability prediction for modern standard arabic. Master’s thesis, Brigham Young University, Provo, UT. find
  16. Nizar Habash. 2010. Introduction to Arabic natural language processing, 1 edition, volume 3 of Synthesis Lectures on Human Language Technologies. Morgan and Claypool Publishers. find
  17. Nizar Habash, Hanada Taha-Thomure, Khalid Elmadani, Zeina Zeino, and Abdallah Abushmaes. 2025. Guidelines for fine-grained sentence-level Arabic readability annotation. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages 359–376, Vienna, Austria. Association for Computational Linguistics.
  18. Reem Hazim, Hind Saddiki, Bashar Alhafni, Muhamed Al Khalil, and Nizar Habash. 2022. Arabic word-level readability visualization for assisted text simplification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 242–249, Abu Dhabi, UAE. Association for Computational Linguistics.
  19. Bruce W. Lee, Yoo Sung Jang, and Jason Lee. 2021. Pushing on text readability assessment: A transformer meets handcrafted linguistic features. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10669– 10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  20. Juan Liberato, Bashar Alhafni, Muhamed Khalil, and Nizar Habash. 2024. Strategies for Arabic readability modeling. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 55–66, Bangkok, Thailand. Association for Computational Linguistics. Slovenia. European Language Resources Association (ELRA).
  21. Wajdi Zaghouani, Nizar Habash, Houda Bouamor, Alla Rozovskaya, Behrang Mohit, Abeer Heider, and Kemal Oflazer. 2015. Correction annotation for non-native Arabic texts: Guidelines and corpus. In Proceedings of the 9th Linguistic Annotation Workshop, pages 129–139, Denver, Colorado, USA. Association for Computational Linguistics.
  22. Matej Martinc, Senja Pollak, and Marko Robnik-Šikonja. 2021. Supervised and unsupervised neural approaches to text readability. Computational Linguistics, 47(1):141–179. ↗
  23. Naoual Nassiri, Violetta Cavalli-Sforza, and Abdelhak Lakhouaja. 2023. Approaches, methods, and resources for assessing the readability of arabic texts. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 22(4). ↗
  24. Hind Saddiki, Nizar Habash, Violetta Cavalli-Sforza, and Muhamed Al Khalil. 2018. Feature optimization for predicting readability of Arabic L1 and L2. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, pages 20–29, Melbourne, Australia. Association for Computational Linguistics. ↗
  25. Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment research. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5366– 5377, Marseille, France. European Language Resources Association. find
  26. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180–189, Portland, Oregon, USA. Association for Computational Linguistics.
  27. Wajdi Zaghouani and Dana Awad. 2016a. Building an arabic punctuated corpus. 2016(1):SSHAPP3148. find
  28. Wajdi Zaghouani and Dana Awad. 2016b. Toward an arabic punctuated corpus: Annotation guidelines and evaluation. In The 2nd Workshop on Arabic Corpora and Processing Tools 2016 Theme: Social Media, volume 22. find
  29. Wajdi Zaghouani, Houda Bouamor, Abdelati Hawwari, Mona Diab, Ossama Obeid, Mahmoud Ghoneim, Sawsan Alqahtani, and Kemal Oflazer. 2016. Guidelines and framework for a large scale Arabic diacritized corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3637–3643, Portorož,

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ibrahim-2025-marsadlab-barec-shared-task-2025/