Elamur Research⌕
☰
home / papers / 2025 / ANLPers at BAREC Shared Task 2025: Readability of Embeddings Training Neural Readability Classifiers on the BAREC Corpus

ANLPers at BAREC Shared Task 2025: Readability of Embeddings Training Neural Readability Classifiers on the BAREC Corpus

Paper · 2025 · Serry Sibaee, Omer Nacar, Yasser Alhabashi et al.

Summary

The authors join the sentence-level Open track of the BAREC Shared Task 2025. The goal is to assign each Arabic sentence one of 19 Taha-Thomure readability levels, ranging from kindergarten to postgraduate difficulty. Their system has two stages. First, each pre-trained Arabic encoder (ARBERTv2, MARBERTv2, AraBERT) produces token vectors, which are mean-pooled into a sentence vector. The vectors from the different encoders are then averaged element-wise into one representation. Second, a deep feed-forward MLP trained with a regression objective maps that vector to a continuous level score. They also tested multilingual sentence encoders and light preprocessing (stripping non-Arabic characters, lemmatizing with Sina Tools), but neither changed results much. The best runs reach QWK of 76.35 on test and 73.00 on blind test, with roughly 45% exact and 61% adjacent accuracy. This is several points below the official baseline, but each run trains in about ten minutes with only 12–15M trainable parameters. The authors also document noise in the BAREC data. The work matters as a cheap, reproducible baseline for fine-grained readability scoring.

Key points

  • Data: CAMeL-Lab/BAREC-Shared-Task-2025-sent, with 54,845 train, 7,310 validation, 7,286 test and 3,420 blind-test sentences. The validation split was folded into training, so the authors themselves flag that the results are not directly comparable with other teams.

  • Encoders tried: ARBERTv2, MARBERTv2, AraBERTv2, LaBSE, all-MiniLM-L6-v2 and a Matryoshka-based model. The best runs used ARBERTv2 (Exp-1) and MARBERTv2 (Exp-2), each with 768-dimensional input.

  • MLP head: shrinking hidden layers such as [4096, 2048, 1024(, 512)], AdamW, dropout, weight decay, early stopping, 800–2000 epochs, batches up to 65,536 on an A100-80GB, and seed 42.

  • Results: Exp-1 scored QWK 76.35, accuracy 44.73% and adjacent accuracy 61.35%. Exp-2 (blind test) scored QWK 73.00, accuracy 44.70% and adjacent accuracy 61.40%. Both are within about 8% of the BAREC baseline.

  • Data problems found:

    • uneven diacritization;
    • erratic punctuation;
    • fragmentary or noisy sentences;
    • questionable labels, especially for poetry and literary excerpts.

    Rare high levels were supplemented with extra Arabic poems and didactic verse (manzomat).

Interesting ideas

  • I1. Freezing the encoders and training only a small MLP head on pooled vectors makes each experiment take minutes, so many configurations can be swept cheaply.
  • I2. Averaging sentence vectors from several independently trained encoders smooths out the quirks of any single model.
  • I3. Ordinal labels are modeled as a continuous regression target and evaluated with QWK and adjacent accuracy, so near-misses are counted separately from gross errors.
  • I4. Toggling lemmatization and normalization on and off showed that the preprocessing barely mattered for this task.
  • I5. A manual error audit of the corpus revealed systematic label noise concentrated in specific genres, here poetry and literary fragments.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): A frozen-feature probe like this could quickly test whether a candidate representation carries a signal before committing to full parser training.
  • I2 (learn): This parallels the witness-parser idea and suggests combining several models' scores or features before deciding on silver labels, rather than trusting one model.
  • I3 (answer): School-course readers could grade passage difficulty or answer quality on an ordinal scale and report off-by-one tolerance alongside exact match.
  • I4 (tokens): Each normalization step in crate en should be ablated against downstream metrics, rather than assumed to be helpful.
  • I5 (tree): When treebanks or converted UD dialects are used, auditing disagreements by genre could expose annotation inconsistencies that cap parser accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Prince Sultan UniversityRiyadh · Saudi Arabia17 works · 120 citations🎓Tuwaiq AcademyRiyadh · Saudi Arabia11 works · 7 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Serry Sibaee, Omer Nacar, Yasser Alhabashi, Adel Ammar and Wadii Boulila. 2025. ANLPers at BAREC Shared Task 2025: Readability of Embeddings Training Neural Readability Classifiers on the BAREC Corpus. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.38

@inproceedings{sibaee2025anlpers,
  title = {{ANLPers at BAREC Shared Task 2025: Readability of Embeddings Training Neural Readability Classifiers on the BAREC Corpus}},
  author = {Serry Sibaee, Omer Nacar, Yasser Alhabashi, Adel Ammar and Wadii Boulila},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.38},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.38}
}

References

18 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. sample) on the link https://huggingface.co/datasets/ JadwalAlmaa/Expand_BAREC. Note: we did not used this new dataset in our training. ↗
  2. Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.
  3. Omer Nacar, Anis Koubaa, Serry Sibaee, Yasser Al-Habashi, Adel Ammar, and Wadii Boulila. 2025. Gate: General arabic text embedding for enhanced semantic textual similarity with matryoshka representation learning and hybrid loss training. arXiv preprint arXiv:2505.24581. ↗
  4. Bashar Alhafni, Reem Hazim, Juan David Pineros Liberato, Muhamed Al Khalil, and Nizar Habash. 2025. The SAMER Arabic text simplification corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16079–16093, Torino, Italia. ELRA and ICCL.
  5. Carolyn R Ransford-Kaldon, E Sutton Flynt, Cristin L Ross, Louis Franceschini, Todd Zoblotsky, Ying Huang, and Brenda Gallagher. 2010. Implementation of effective intervention: An empirical study to evaluate the efficacy of fountas & pinnell’s leveled literacy intervention system (lli). 2009-2010. Center for Research in Educational Policy (CREP). find
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  7. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  8. Khalid N. Elmadani, Bashar Alhafni, Hanada Taha, and Nizar Habash. 2025a. BAREC shared task 2025 on Arabic readability assessment. In Proceedings of the Third Arabic Natural Language Processing Conference, Suzhou, China. Association for Computational Linguistics.
  9. Khalid N. Elmadani, Nizar Habash, and Hanada Taha-Thomure. 2025b. A large and balanced corpus for fine-grained Arabic readability assessment. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16376–16400, Vienna, Austria. Association for Computational Linguistics.
  10. Nizar Habash, Hanada Taha-Thomure, Khalid Elmadani, Zeina Zeino, and Abdallah Abushmaes. 2025. Guidelines for fine-grained sentence-level Arabic readability annotation. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages 359–376, Vienna, Austria. Association for Computational Linguistics.
  11. Tymaa Hammouda, Mustafa Jarrar, and Mohammed Khalilia. 2024. Sinatools: Open source toolkit for arabic natural language processing. Preprint, arXiv:2411.01523. ↗
  12. Reem Hazim, Hind Saddiki, Bashar Alhafni, Muhamed Al Khalil, and Nizar Habash. 2022. Arabic word-level readability visualization for assisted text simplification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 242–249, Abu Dhabi, UAE. Association for Computational Linguistics.
  13. Juan Liberato, Bashar Alhafni, Muhamed Khalil, and Nizar Habash. 2024. Strategies for Arabic readability modeling. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 55–66, Bangkok, Thailand. Association for Computational Linguistics.
  14. Serry Sibaee, Abdullah Alharbi, Samar Ahmad, Omer Nacar, Anis Koubaa, and Lahouari Ghouti. 2024. ASOS at KSAA-CAD 2024: One embedding is all you need for your dictionary. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 697–703, Bangkok, Thailand. Association for Computational Linguistics.
  15. Serry Sibaee, Omer Nacar, Adel Ammar, Yasser Al-Habashi, Abdulrahman Al-Batati, and Wadii Boulila. 2025. From guidelines to practice: A new paradigm for arabic language model evaluation. Preprint, arXiv:2506.01920. ↗
  16. Hanada Taha-Thomure. 2017. Arabic Language Text  ñJ‚Ó J ’JË é£ @XA Jë Q K AªÓ Leveling ( ñ’ JË@ HAK éJ  K. QªË@ ). Educational  ( ©K PñJË@ð Qå„ JÊË ø ñK. Q Ë@ H. AJºË@ P@X ) . Book House Suna-Şeyma Uçar, Itziar Aldabe, Nora Aranberri, and Ana Arruarte. 2024. Exploring automatic readability assessment for science documents within a multilingual educational context. International Journal of Artificial Intelligence in Education, 34(4):1417– 1459. find
  17. Giulia Venturi, Tommaso Bellandi, Felice Dell’Orletta, and Simonetta Montemagni. 2015. NLP–based readability assessment of health–related texts: a case study on Italian informed consent forms. In Proceedings of the Sixth International Workshop on Health Text Mining and Information Analysis, pages 131– 141, Lisbon, Portugal. Association for Computational Linguistics. find
  18. Hamid Mohammadi, Seyed Hossein Khasteh, Tahereh Firoozi, and Taha Samavati. 2023. Text as environment: A deep reinforcement learning text readability assessment model. Preprint, arXiv:1912.05957. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sibaee-2025-anlpers-barec-shared-task-2025/