Elamur Research⌕
☰
home / papers / 2025 / Guidelines for Fine-grained Sentence-level Arabic Readability Annotation

Guidelines for Fine-grained Sentence-level Arabic Readability Annotation

Paper · 2025 · Nizar Habash, Hanada Taha-Thomure, Khalid N. Elmadani et al.

Summary

The paper addresses the lack of a large, fine-grained, manually labeled resource for judging how hard Arabic text is to read. The authors present the annotation guidelines behind BAREC (Balanced Arabic Readability Evaluation Corpus). BAREC contains roughly 69K sentences across many genres, each assigned one of 19 reading levels running from kindergarten to postgraduate. The guidelines adapt the Taha/Arabi21 book-leveling framework to single sentences. Native-speaker educators refined them over several rounds of training. Annotators score how easily an independent reader can pronounce a sentence and grasp its literal meaning. They consider vocabulary, morphology, syntax and content rather than surface length. Agreement among annotators was high in the final phase. A fine-tuned AraBERTv02 baseline comes close to human agreement on QWK, though its exact accuracy is lower. The 19 levels collapse cleanly into 7-, 5- and 3-level schemes, so the resource serves both detailed pedagogy and coarser NLP uses. The corpus and guidelines are publicly released.

Key points

  • BAREC has 69,441 sentences (1M+ words) labeled on 19 levels named in Abjad order (1-alif … 19-qaf). These map deterministically to 7, 5 and 3 coarser levels; for example, 11-kaf becomes 4/7, 2/5 and 1/3.
  • The guidelines are based on the Taha/Arabi21 framework, which had already been used to level more than 9,000 children's books.
  • Inter-annotator agreement in Phase 2 reached a Quadratic Weighted Kappa of 81.8%.
  • The baseline fine-tunes AraBERTv02 for 3 epochs on a V100 (learning rate 5e-5, batch size 64). With the full training data it reached:
    • 55.8% accuracy on 19 levels;
    • QWK of 81.0%;
    • 64.9%, 69.1% and 74.7% accuracy on 7, 5 and 3 levels;
    • an average level distance of 1.09.
  • Planned extensions include 19-level readability lexicons, vocabulary leveling, syntactic treebanks, and a tool that helps children's authors write to a target level.

Interesting ideas

  • I1. The authors annotate only at the finest granularity and derive every coarser scheme by a fixed, nested mapping, so one labeling effort supports many task resolutions.
  • I2. Annotators begin at the easiest level and move upward only when they find a concrete trigger (a word, a morphological form, a construction or a content feature), which makes each label traceable to a reason.
  • I3. When a form is ambiguous, the guidelines tell annotators to adopt the simpler reading and to ignore optional diacritics, which removes a major source of disagreement.
  • I4. Model quality is reported against human inter-annotator agreement on several metrics (exact accuracy, ±1 accuracy, mean distance, QWK), which shows that the model nearly matches humans on ordinal agreement while lagging on exact hits.
  • I5. A learning curve over 1/8, 1/4, 1/2 and the full training set shows steady gains, indicating whether more labeled data is still worth collecting.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-course test items could be graded on one fine difficulty scale that collapses to grade bands, keeping reader evaluation consistent across granularities.
  • I2 (world): A difficulty or complexity score for a text could be computed the same way, as the maximum over explicit feature triggers found in the UD tree and the world model, so the score stays explainable.
  • I3 (tokens): When the tagger or lemmatizer faces a genuinely ambiguous form with no disambiguating context, a "prefer the commoner reading" default could be logged as an explicit, inspectable rule.
  • I4 (learn): Evaluations of ordinal outputs, such as confidence tiers or grade levels, could use the witness-parser or teacher agreement rate as the ceiling and include distance-aware metrics, not just accuracy.
  • I5 (learn): Running the same fractional-data curve on silver treebanks would show whether adding more silver data or improving its quality pays off more for the gparse parser.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksEducation · 203 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Nizar Habash, Hanada Taha-Thomure, Khalid N. Elmadani, Zeina Zeino and Abdallah Abushmaes. 2025. Guidelines for Fine-grained Sentence-level Arabic Readability Annotation. Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.law-1.30

@inproceedings{habash2025guidelines,
  title = {{Guidelines for Fine-grained Sentence-level Arabic Readability Annotation}},
  author = {Nizar Habash, Hanada Taha-Thomure, Khalid N. Elmadani, Zeina Zeino and Abdallah Abushmaes},
  booktitle = {Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025)},
  year = {2025},
  doi = {10.18653/v1/2025.law-1.30},
  url = {https://doi.org/10.18653/v1/2025.law-1.30}
}

References

54 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 11–16, San Diego, California.
  2. Abbas Mahmoud Al-Akkad. 1938. Sarah. Hindawi. Imam Muhammad al Bukhari. 846. Sahih al-Bukhari. Dar Ibn Khathir. find
  3. M Al-Dawsari. 2004. The assessment of readability books content (boys-girls) of the first grade of intermediate school according to readability standards. Sultan Qaboos University, Muscat. find
  4. Hend S Al-Khalifa and Amani A Al-Ajlan. 2010. Automatic readability measurements of the Arabic text: An exploratory study. Arabian Journal for Science and Engineering, 35(2 C):103–124. find
  5. Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.
  6. Bayan Al-Safadi. 2005. lection of poetry and Al-Kashkoul: se-prose for children Al-Sa’ih A. Alfaifi. 2015. Building the Arabic Learner Corpus and a System for Arabic Error Annotation. Ph.D. thesis, University of Leeds. find
  7. Bashar Alhafni, Reem Hazim, Juan David Pineros Liberato, Muhamed Al Khalil, and Nizar Habash. 2024. The SAMER Arabic text simplification corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16079–16093, Torino, Italia. ELRA and ICCL.
  8. Richard L Allington, Kimberly McCuiston, and Monica Billen. 2015. What research says about text complexity and learning to read. The Reading Teacher, 68(7):491–501. ↗
  9. Shatha Altammami, Eric Atwell, and Ammar Alsalka. 2019. The arabic–english parallel corpus of authentic hadith. International Journal on Islamic Applications in Computer Science And Technology-IJASAT. find
  10. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  11. Amelia T. Barber and Susan L. Klauda. 2020. How reading motivation and engagement enable reading achievement: Policy implications. Policy Insights from the Behavioral and Brain Sciences, 7(1):27–34. ↗
  12. May Bashendy, Salam Albatarni, Sohaila Eltanbouly, Eman Zahran, Hamdo Elhuseyin, Tamer Elsayed, Walid Massoud, and Houda Bouamor. 2024. Qaes: First publicly-available trait-specific annotations for automated scoring of arabic essays. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 337–351.
  13. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  14. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213. ↗
  15. Kevyn Collins-Thompson and James P. Callan. 2004. A language modeling approach to predicting reading difficulty. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 193–200, Boston, Massachusetts, USA. Association for Computational Linguistics.
  16. Tovly Deutsch, Masoud Jasbi, and Stuart Shieber. 2020. Linguistic features for readability assessment. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–17, Seattle, WA, USA → Online. Association for Computational Linguistics.
  17. Afrizal Doewes, Nughthoh Arfawi Kurdhi, and Akrati Saxena. 2023. Evaluating quadratic weighted kappa as the standard performance metric for automated essay scoring. In Proceedings of the 16th International Conference on Educational Data Mining, pages 103– 113, Bengaluru, India. International Educational Data Mining Society. find
  18. William H DuBay. 2004. The principles of readability. Online Submission. find
  19. Kais Dukes, Eric Atwell, and Nizar Habash. 2013. Supervised collaboration for syntactic annotation of quranic arabic. Language resources and evaluation, 47(1):33–62. find
  20. Matthias Eck and Chiori Hori. 2005. Overview of the IWSLT 2005 evaluation campaign. In Proceedings of the Second International Workshop on Spoken Language Translation, Pittsburgh, Pennsylvania, USA. find
  21. Mahmoud El-Haj and Paul Rayson. 2016. OSMAN — a novel Arabic readability metric. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC‘16), pages 250–255, Portorož, Slovenia. European Language Resources Association (ELRA). find
  22. Mo El-Haj, Sultan Almujaiwel, Damith Premasiri, Tharindu Ranasinghe, and Ruslan Mitkov. 2024. DARES: Dataset for Arabic readability estimation of school materials. In Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context @ LREC-COLING 2024, pages 103–113, Torino, Italia. ELRA and ICCL.
  23. Mo El-Haj and Saad Ezzini. 2024. The multilingual corpus of world‘s constitutions (MCWC). In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024, pages 57–66, Torino, Italia. ELRA and ICCL. find
  24. Khalid N. Elmadani, Nizar Habash, and Hanada Taha-Thomure. 2025. A large and balanced corpus for fine-grained Arabic readability assessment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), Vienna, Austria. Association for Computational Linguistics.
  25. Lijun Feng, Martin Jansche, Matt Huenerfauth, and Noémie Elhadad. 2010. A comparison of features for automatic readability assessment. In Coling 2010: Posters, pages 276–284, Beijing, China. Coling 2010 Organizing Committee.
  26. Jonathan Forsyth. 2014. Automatic readability prediction for modern standard Arabic. In Proceedings of the Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT). find
  27. Irene C Fountas and Gay Su Pinnell. 2006. Leveled books (k-8): Matching texts to readers for effective teaching. Heinemann Educational Books. find
  28. Nizar Habash, Muhammed AbuOdeh, Dima Taji, Reem Faraj, Jamila El Gizuli, and Omar Kallas. 2022. Camel treebank: An open multi-genre Arabic dependency treebank. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2672–2681, Marseille, France. European Language Resources Association.
  29. Nizar Habash and David Palfreyman. 2022. ZAEBUC: An annotated Arabic-English bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79–88, Marseille, France. European Language Resources Association. find
  30. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, pages 15–22. Springer, Netherlands. ↗
  31. Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. find
  32. Muhamed Al Khalil, Hind Saddiki, Nizar Habash, and Latifa Alfalasi. 2018. A Leveled Reading Corpus of Modern Standard Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan.
  33. Adam Kilgarriff, Frieda Charalabopoulou, Maria Gavrilidou, Janne Bondi Johannessen, Saussan Khalil, Sofie Johansson Kokkinakis, Robert Lew, Serge Sharoff, Ravikiran Vadlapudi, and Elena Volodina. 2014. Corpus-based vocabulary lists for language learners for nine languages. Language Resources and Evaluation, 48(1):121–163. ↗
  34. G.R. Klare. 1963. The Measurement of Readability. Iowa State University Press. find
  35. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics. ↗
  36. Bruce W. Lee, Yoo Sung Jang, and Jason Lee. 2021. Pushing on text readability assessment: A transformer meets handcrafted linguistic features. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10669– 10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  37. Juan Liberato, Bashar Alhafni, Muhamed Khalil, and Nizar Habash. 2024. Strategies for Arabic readability modeling. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 55–66, Bangkok, Thailand. Association for Computational Linguistics.
  38. Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the Language Resources and Evaluation Conference (LREC), Portorož, Slovenia.
  39. Farah Nadeem and Mari Ostendorf. 2018. Estimating linguistic complexity for science texts. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 45–55, New Orleans, Louisiana. Association for Computational Linguistics. find
  40. Tarek Naous, Michael J Ryan, Anton Lavrouk, Mohit Chandra, and Wei Xu. 2024. ReadMe++: Benchmarking multilingual language models for multi-domain readability assessment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12230–12266, Miami, Florida, USA. Association for Computational Linguistics. find
  41. Naoual Nassiri, Violetta Cavalli-Sforza, and Abdelhak Lakhouaja. 2023. Approaches, methods, and resources for assessing the readability of arabic texts. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 22(4). ↗
  42. Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 186–195, Honolulu, Hawaii. Association for Computational Linguistics.
  43. Hind Saddiki, Nizar Habash, Violetta Cavalli-Sforza, and Muhamed Al Khalil. 2018. Feature optimization for predicting readability of Arabic l1 and l2. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, pages 20–29. ↗
  44. Eli Smith and Cornelius Van Dyck. 1860. New Testament (Arabic Translation). find
  45. Eli Smith and Cornelius Van Dyck. 1865. Old Testament (Arabic Translation). find
  46. Rasha Soliman and Laila Familiar. 2024. Creating a CEFR Arabic vocabulary profile: A frequency-based multi-dialectal approach. Critical Multilingualism Studies, 11(1):266–286. ↗
  47. Hanada Taha-Thomure. 2007. Poems and News ( PAJ. k @ð PAª ƒ  @ ). Educational Book House Resources and Evaluation Conference, pages 5366– 5377, Marseille, France. European Language Resources Association. find
  48. Sowmya Vajjala and Ivana Lučić. 2018. OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 297–304, New Orleans, Louisiana. Association for Computational Linguistics.
  49. Sowmya Vajjala and Detmar Meurers. 2012. On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 163–173, Montréal, Canada. Association for Computational Linguistics.
  50. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗
  51. Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2016. Text readability assessment for second language learners. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 12–22, San Diego, CA. Association for Computational Linguistics.
  52. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297.  ( ©K PñJË@ð Qå„ JÊË ø ñK. Q Ë@ H. AJºË@ P@X ) . ↗
  53. Hanada Taha-Thomure. 2017. Arabic Language Text  ñJ‚Ó J ’JË é£ @XA Jë Q K AªÓ Leveling ( ñ’ JË@ HAK Educational éJ  K. QªË@ ).  ( ©K PñJË@ð Qå„ JÊË ø ñK. Q Ë@ H. AJºË@ P@X ) . Book House Toshiyuki Takezawa, Genichiro Kikui, Masahide Mizushima, and Eiichiro Sumita. 2007. Multilingual spoken language corpus development for communication research. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 12, Number 3, September 2007: Special Issue on Invited Papers from ISCSLP 2006, pages 303–324. Ibn Tufail. 1150. Hayy ibn Yaqdhan. Hindawi. Unknown. 12th century. One Thousand and One Nights. find
  54. Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment research. In Proceedings of the Thirteenth Language B AREC Annotation Guidelines Cheat Sheet and Annotation Interface find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/habash-2025-guidelines-fine-grained-sentence-level/