Elamur Research⌕
☰
home / papers / 2025 / PalNLP at BAREC Shared Task 2025: Predicting Arabic Readability Using Ordinal Regression and K-Fold Ensemble Learning

PalNLP at BAREC Shared Task 2025: Predicting Arabic Readability Using Ordinal Regression and K-Fold Ensemble Learning

Paper · 2025 · Mutaz Ayesh

Summary

The paper tackles sentence-level readability prediction for Arabic on BAREC, a corpus of over 69 thousand texts graded on a 19-level scale adapted from the Taha/Arabi21 framework. The author competed in the strict track, so no outside data was used. Because the levels are ordered, the system treats them as a continuous target rather than as independent classes. It starts from CAMeL-Lab's readability-arabertv2-d3tok-CE model, which was originally trained with cross-entropy, and refits it with a single regression output trained on MSE. Continuous scores become levels through cut-points tuned to maximize Quadratic Weighted Kappa (QWK). The official training and development data were pooled into 5 stratified folds, rare levels were oversampled, and the best model from each fold was combined into an ensemble. The system scored 81.1 QWK on the blind test, ranking 18th of 24 and just below the 81.5 baseline. Its ±1 accuracy was competitive, but its exact-level accuracy was weak. Error analysis traced many large mistakes to very short sentences and to diacritized text.

Key points

  • Model and training. The base model is readability-arabertv2-d3tok-CE with a regression head. Training used MSE loss, AdamW, a 2e-5 learning rate, batch size 16, and inputs padded to 512 tokens. Training ran up to 6 epochs, with early stopping (patience 3) and a scheduler combining linear warmup with ReduceLROnPlateau keyed to validation QWK.

  • Thresholds and class balance. Multi-pass grid coordinate descent tuned the class thresholds, giving roughly 1–2% over plain rounding. Rare levels were oversampled with PyTorch's WeightedRandomSampler using inverse-frequency weights.

  • Internal results. Cross-validation QWK stayed between about 79.85 and 80.21, and the provided test set reached 77.7. When diacritics were removed with PyArabic's strip_diacritics, the test set rose to 81.25 QWK. Exact accuracy went from 29.99% to 34.37%, ±1 accuracy from 65.88% to 69.48%, and the worst-error range narrowed from (-15, 12) to (-11, 8).

  • Blind test versus baseline.

    MetricPalNLPBaseline
    QWK81.181.5
    Exact accuracy33.1%58.1%
    ±1 accuracy69.8%72.0%
    3-class accuracy72.5%76.5%

    The gap to the baseline shrinks steadily as the label scale gets coarser.

  • Error analysis. The author examined the 100 most-wrong predictions, 50 in each direction. Among underestimated items, 94% had fewer than 5 words, and many came from specialized sources such as the Emirati curriculum and the Quran. In one case, a diacritized proper name with gold level 3 was predicted as level 15.

Interesting ideas

  • I1. Fitting an ordered label as a regression score and then learning per-class cut-points by coordinate descent on the target metric beats simple rounding at no extra model cost.
  • I2. Reporting exact, ±1, and collapsed 7/5/3-class accuracy side by side shows whether a model knows the broad picture but fails at fine distinctions.
  • I3. Inspecting the worst errors separately for over- and under-estimation exposed a superficial cue: diacritics made texts look hard. Removing that cue at inference gave a large gain.
  • I4. Each fold keeps its own tuned thresholds, and fold models are ensembled after their own calibration rather than with one global set of cut-points.
  • I5. When the official dev split stops being informative, pooling train and dev into stratified k-folds gives steadier model selection, though results become less comparable with others.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): A school-course reader could grade passages or questions on an ordered difficulty scale this way, tuning the cut-points on held-out tests instead of rounding.
  • I2 (learn): Evaluation of ordinal outputs, such as confidence tiers or level estimates, could always include coarsened views to locate where errors actually live.
  • I3 (learn): Mova could add negative controls that strip or perturb surface marks (casing, quotes, punctuation) to check that decisions rest on structure, not orthographic noise.
  • I4 (learn): When several witness models are trained on different splits, calibrating each one separately before voting may produce cleaner silver labels.
  • I5 (tree): Small English treebanks for gparse could use stratified folds for checkpoint selection, while keeping the official test split untouched for honest reporting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cardiff UniversityCardiff · United Kingdom23 works · 2,549 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Mutaz Ayesh. 2025. PalNLP at BAREC Shared Task 2025: Predicting Arabic Readability Using Ordinal Regression and K-Fold Ensemble Learning. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.47

@inproceedings{ayesh2025palnlp,
  title = {{PalNLP at BAREC Shared Task 2025: Predicting Arabic Readability Using Ordinal Regression and K-Fold Ensemble Learning}},
  author = {Mutaz Ayesh},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.47},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.47}
}

References

14 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Reem Hazim, Hind Saddiki, Bashar Alhafni, Muhamed Al Khalil, and Nizar Habash. 2022. Arabic word-level readability visualization for assisted text simplification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 242–249, Abu Dhabi, UAE. Association for Computational Linguistics.
  2. Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.
  3. Juan Liberato, Bashar Alhafni, Muhamed Khalil, and Nizar Habash. 2024. Strategies for Arabic readability modeling. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 55–66, Bangkok, Thailand. Association for Computational Linguistics.
  4. Bashar Alhafni, Reem Hazim, Juan David Pineros Liberato, Muhamed Al Khalil, and Nizar Habash. 2024. The SAMER Arabic text simplification corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16079–16093, Torino, Italia. ELRA and ICCL.
  5. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2022. Readability controllable biomedical document summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4667–4680, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  6. Mutaz Ayesh, Nicolás Gutiérrez-Rolón, and Fernando Alva-Manchego. 2025. CardiffNLP at CLEARS-2025: Prompting large language models for plain language and easy-to-read text rewriting. ↗
  7. Margot Madina, Itziar Gonzalez-Dios, and Melanie Siegel. 2024. Towards reliable e2r texts: A proposal for standardized evaluation practices. In Computers Helping People with Special Needs, pages 224–231, Cham. Springer Nature Switzerland. find
  8. Paloma Martínez, Lourdes Moreno, and Alberto Ramos. 2024. Exploring large language models to generate easy to read content. Preprint, arXiv:2407.20046. ↗
  9. Beatriz Botella-Gil, Isabel Espinosa-Zaragoza, Alba Bonet-Jover, Margot Madina, Lucas Molino Piñar, Paloma Moreda, Itziar Gonzalez-Dios, María Teresa Martín Valdivia, and Ureña. 2025. Overview of CLEARS at IberLEF 2025: Challenge for Plain Language and Easy-to-Read Adaptation for Spanish Texts. Procesamiento del Lenguaje Natural, 75. find
  10. Laila Suwaiyan. 2018. Diglossia in the arabic language. International Journal of Language Linguistics, 5. ↗
  11. Khalid N. Elmadani, Bashar Alhafni, Hanada Taha, and Nizar Habash. 2025a. BAREC shared task 2025 on Arabic readability assessment. In Proceedings of the Third Arabic Natural Language Processing Conference, Suzhou, China. Association for Computational Linguistics.
  12. Hanada Taha-Thomure. 2017. Arabic Language Text  ñJ‚Ó J ’JË é£ @XA Jë Q K AªÓ Leveling ñ’ JË@ HAK Khalid N. Elmadani, Nizar Habash, and Hanada Taha-Thomure. 2025b. A Large and Balanced Corpus for Fine-grained Arabic Readability Assessment. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16376–16400, Vienna, Austria. Association for Computational Linguistics.
  13. Isabel Espinosa-Zaragoza, José Abreu-Salas, Paloma Moreda, and Manuel Palomar. 2023. Automatic text simplification for people with cognitive disabilities: Resource creation within the ClearText project. In Proceedings of the Second Workshop on Text Simplification, Accessibility and Readability, pages 68–77, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. find
  14. Nizar Habash, Hanada Taha-Thomure, Khalid N. Elmadani, Zeina Zeino, and Abdallah Abushmaes. 2025. Guidelines for fine-grained sentence-level Arabic readability annotation. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages 359–376, Vienna, Austria. Association for Computational Linguistics. é  J K. Q  ª Ë@ . Educational Book House H A J º Ë@ P@X .   ©K Pñ JË@ð Qå „ JÊË ø ñK. Q Ë@ .

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ayesh-2025-palnlp-barec-shared-task-2025/