Elamur Research⌕
☰
home / papers / 2025 / AMAR at BAREC Shared Task 2025: Arabic Meta-learner for Assessing Readability

AMAR at BAREC Shared Task 2025: Arabic Meta-learner for Assessing Readability

Paper · 2025 · Mostafa Saeed, Rana Waly and Abdelaziz Ashraf Hussein

Summary

The paper describes the AMAR team's entry to the BAREC Shared Task 2025. The task is to grade Arabic text on a 19-level readability scale, at sentence and document granularity, under three tracks with different data restrictions. Arabic is hard here because of its rich morphology and because BAREC mixes many genres. The team fine-tuned seven Arabic and multilingual transformer encoders as regressors, after stripping diacritics and kashida. They then fed the encoders' outputs into a lightweight second-stage learner. In the Constrained track they also added word-difficulty statistics from the SAMER lexicon. In the Open track they tested GPT-4.1 prompted with the full annotation guidelines. A CategoricalNB meta-learner reached 86.4% QWK on the blind sentence-level test and placed 2nd in Tracks 1 and 2. The approach did poorly at document level, the lexicon added almost nothing, and the LLM was weak. The study shows that cheap stacking over diverse encoders reliably beats the best single model for fine-grained ordinal labels.

Key points

  • BAREC holds 69,441 manually labelled sentences (over a million words) across 19 levels, from kindergarten to postgraduate. It is split into train, dev, public test, and a blind test of 3,420 sentences.
  • The base models were AraBERTv02, AraBERTv2, MARBERTv2, CAMeLBERT-MSA, XLM-RoBERTa large, bert-qarib and NuSentiment multilingual. CAMeLBERT-MSA was the strongest, with 82.8% dev QWK and 83.8% public-test QWK.
  • The meta-learner inputs were each model's prediction, the mean prediction, and the mean top-class probability. Gaussian and Categorical Naive Bayes beat logistic and linear regression, random forests and bagging, reaching about 84.9% public-test QWK.
  • Blind-test results were 86.4% QWK with roughly 40% accuracy (39.7% in Track 1, 39.9% in Track 2). At document level, scoring each document by its maximum sentence level gave only 69.6% QWK and 34% accuracy, which placed 7th.
  • SAMER features (the mode, maximum and mean lemma level per sentence) improved accuracy by only 0.2%. Words were lemmatized with CAMeL Tools, with an edit-distance fallback for missing entries. GPT-4.1 given the guidelines scored just 40.7% QWK on dev.

Interesting ideas

  • I1. The second-stage learner works only on the base models' outputs, plus their mean and average confidence, so combining models stays cheap and transparent. Naive Bayes won over heavier learners in this setting.
  • I2. Treating ordinal labels as regression raised QWK but kept exact-match accuracy around 40%, and that gap hurt document-level results. A single ordinal metric can hide this weakness.
  • I3. Each lemma is looked up in a graded lexicon, with an edit-distance fallback when it is missing. The per-word levels are then summarised as max, mode and mean to give a sentence-level signal.
  • I4. Rolling sentence scores up to documents by simply taking the maximum performed badly. This suggests document-level properties need their own modelling rather than a single extreme sentence.
  • I5. An LLM given the full human annotation guidelines fell far short of fine-tuned encoders. Rich instructions alone did not reproduce expert labelling.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When several witness parsers vote on silver data, a small learner can be trained on their outputs, mean and confidence to decide which arc or label to accept, instead of using a fixed majority rule.
  • I2 (learn): For graded outputs such as difficulty or confidence levels, evaluations should always report both an exact-match score and a distance-aware score.
  • I3 (tokens): Lemma lookups against WordNet-style resources could use a controlled nearest-match fallback that flags the match as approximate, so the explanation trace records the guess.
  • I4 (world): When summarising a text-level state, such as overall tone or complexity, Mova should aggregate over its entities and events instead of letting one outlier sentence decide.
  • I5 (learn): LLM teacher tiers should be benchmarked against a gold sample before their labels feed training, with the prompted-LLM score kept as a baseline to beat.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations🎓Özyeğin UniversityIstanbul · Türkiye3 works · 17 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Mostafa Saeed, Rana Waly and Abdelaziz Ashraf Hussein. 2025. AMAR at BAREC Shared Task 2025: Arabic Meta-learner for Assessing Readability. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.45

@inproceedings{saeed2025amar,
  title = {{AMAR at BAREC Shared Task 2025: Arabic Meta-learner for Assessing Readability}},
  author = {Mostafa Saeed, Rana Waly and Abdelaziz Ashraf Hussein},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.45},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.45}
}

References

31 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.
  4. Bashar Alhafni, Reem Hazim, Juan David Pineros Liberato, Muhamed Al Khalil, and Nizar Habash. 2024. The SAMER Arabic text simplification corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16079–16093, Torino, Italia. ELRA and ICCL.
  5. Bharat Ram Ambati, Siva Reddy, and Mark Steedman. 2016. Assessing relative sentence complexity using an incremental CCG parser. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1051–1057, San Diego, California. Association for Computational Linguistics. find
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. ↗
  8. Edgar Dale and Jeanne S Chall. 1948. A formula for predicting readability: Instructions. Educational research bulletin, pages 37–54. find
  9. Tovly Deutsch, Masoud Jasbi, and Stuart Shieber. 2020. Linguistic features for readability assessment. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–17, Seattle, WA, USA → Online. Association for Computational Linguistics.
  10. Khalid Elmadani, Nizar Habash, and Hanada Taha. 2025a. A large and balanced corpus for fine-grained Arabic readability assessment. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16376–16400, Vienna, Austria. Association for Computational Linguistics.
  11. Khalid N. Elmadani, Bashar Alhafni, Hanada Taha, and Nizar Habash. 2025b. BAREC shared task 2025 on Arabic readability assessment. In Proceedings of the Third Arabic Natural Language Processing Conference, Suzhou, China. Association for Computational Linguistics.
  12. Dengzhao Fang, Jipeng Qiang, Yi Zhu, Yunhao Yuan, Wei Li, and Yan Liu. 2025. Progressive document-level text simplification via large language models. arXiv preprint arXiv:2501.03857. ↗
  13. Ibnu Fitrianto, Cahya Edi Setyawan, and Malikus Saleh. 2024. Utilizing artificial intelligence for personalized arabic language learning plans. International Journal of Post Axial: Futuristic Teaching and Learning, pages 30–40. find
  14. Rudolf Franz Flesch and Alan J Gould. 1949. The art of readable writing. (No Title). find
  15. Nizar Habash, Hanada Taha-Thomure, Khalid Elmadani, Zeina Zeino, and Abdallah Abushmaes. 2025. Guidelines for fine-grained sentence-level Arabic readability annotation. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages 359–376, Vienna, Austria. Association for Computational Linguistics.
  16. Reem Hazim, Hind Saddiki, Bashar Alhafni, Muhamed Al Khalil, and Nizar Habash. 2022. Arabic word-level readability visualization for assisted text simplification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 242–249, Abu Dhabi, UAE. Association for Computational Linguistics.
  17. Chieh-Yang Huang, Jing Wei, and Ting-Hao Kenneth Huang. 2024. Generating educational materials with different levels of readability using llms. In Proceedings of the Third Workshop on Intelligent and Interactive Writing Assistants, pages 16–22. find
  18. Joseph Marvin Imperial and Ekaterina Kochmar. 2023. Automatic readability assessment for closely related languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5371–5386, Toronto, Canada. Association for Computational Linguistics. ↗
  19. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  20. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report. ↗
  21. Justin Lee and Sowmya Vajjala. 2022. A neural pairwise ranking model for readability assessment. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3802–3813, Dublin, Ireland. Association for Computational Linguistics. ↗
  22. Arabic automated essay scoring with synthetic data and error injection. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 549–563, Vienna, Austria. Association for Computational Linguistics. find
  23. Hind Saddiki, Nizar Habash, Violetta Cavalli-Sforza, and Muhamed Al Khalil. 2018. Feature optimization for predicting readability of Arabic L1 and L2. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, pages 20–29, Melbourne, Australia. Association for Computational Linguistics. ↗
  24. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. ↗
  25. Nasser Zalmout, Hind Saddiki, and Nizar Habash. 2016. Analysis of foreign language teaching methods: An automatic readability approach. In Proceedings of the 3rd Workshop on Natural Language Processing Techniques for Educational Applications (NLPTEA2016), pages 122–130, Osaka, Japan. find
  26. The COLING 2016 Organizing Committee. find
  27. Juan Liberato, Bashar Alhafni, Muhamed Khalil, and Nizar Habash. 2024. Strategies for Arabic readability modeling. In Proceedings of the Second Arabic Natural Language Processing Conference, pages 55–66, Bangkok, Thailand. Association for Computational Linguistics.
  28. Fiammetta Marulli, Lelio Campanile, Maria Stella de Biase, Stefano Marrone, Laura Verde, and Marianna Bifulco. 2024. Understanding readability of large language models output: an empirical analysis. Procedia Computer Science, 246:5273–5282. find
  29. Tarek Naous, Michael J Ryan, Anton Lavrouk, Mohit Chandra, and Wei Xu. 2024. ReadMe++: Benchmarking multilingual language models for multi-domain readability assessment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12230–12266, Miami, Florida, USA. Association for Computational Linguistics. find
  30. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  31. Chatrine Qwaider, Bashar Alhafni, Kirill Chirkunov, Nizar Habash, and Ted Briscoe. 2025. Enhancing find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saeed-2025-amar-barec-shared-task-2025/