Elamur Research⌕
☰
home / papers / 2024 / DARES: Dataset for Arabic Readability Estimation of School Materials

DARES: Dataset for Arabic Readability Estimation of School Materials

Paper · 2024 · Mo El-Haj, Sultan Almujaiwel, Damith Premasiri et al.

Summary

DARES tackles Arabic readability estimation for school materials. The authors argue that earlier Arabic corpora, which sample random passages from textbooks, cannot show whether the explanation of a particular concept suits a given grade. DARES instead collects only concept-centred texts: a target term together with the passage that explains it.

The source is 307 Arabic-language Saudi textbooks from the 2021 curriculum, covering grades 1–12 and 116 subjects. PDFs were converted to text, with Tesseract OCR used for text inside images. An Arabic linguist then post-edited the output, since varied fonts degraded OCR quality.

The resulting 13,335 instances support two tasks:

  • a coarse-grained task that predicts the school stage
  • a fine-grained task that predicts the exact grade

Five Arabic-capable transformers were fine-tuned under several input settings. CAMeLBERTmix with the subject name added to the text worked best.

The resource matters because it is the first concept-aware readability dataset. It can help teachers select or adapt explanations that fit their pupils' level.

Key points

  • DARES has 13,335 instances drawn from 307 books and 7,171,857 words. Grades are grouped into early elementary (1–3), upper elementary (4–6), intermediate (7–9) and high school (10–12).
  • The best model was CAMeLBERTmix in the "Subject + Text" setting:
    • coarse task: weighted F1 0.91, macro F1 0.79
    • fine task: weighted F1 0.68, macro F1 0.55
  • Multilingual models fell behind Arabic-specific ones.
  • In the fine-grained task, 871 of 2,681 test items were misclassified. Most errors fell between neighbouring grades, especially grades 10–12.
  • Subjects with the most errors included Science, Arabic Language, Artistic Education and Islamic Studies; Chemistry and Physics added errors at grade level. The dataset and models are released on GitHub.

Interesting ideas

  • I1. Anchoring each sample to a concept (a term plus its explanation) lets difficulty be judged per idea rather than per random passage, which makes the labels easier to interpret and act on.
  • I2. Prepending the subject name as extra input gave the best scores, showing that cheap domain metadata can disambiguate difficulty.
  • I3. Defining two label granularities over the same data (stage vs. exact grade) separates easy distinctions from hard ones and shows where models break.
  • I4. Reporting weighted alongside macro F1 exposes how class imbalance flatters a model; here the gap was large in both tasks.
  • I5. Using OCR for bulk extraction and a linguist for targeted correction is a pragmatic way to build a clean corpus from messy PDFs.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Concept-plus-gloss pairs graded by school year could attach an "expected grade" attribute to ontology entries, so explanations of a concept can be chosen for the reader's level.
  • I2 (answer): School-course readers could condition on the subject tag of a test text to calibrate which vocabulary and constructions are expected at that grade.
  • I3 (learn): Evaluations could report both coarse and fine scores, checking whether errors stay inside the correct band, as a sanity check against misleading single numbers.
  • I4 (learn): Evaluation reports could always pair frequency-weighted and per-class averages, so rare grades or rare constructions are not hidden by common ones.
  • I5 (tokens): Textbook-derived English test material could go through automatic extraction followed by a focused human pass, keeping tokenizer and tagger inputs free of conversion noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 worksEducation · 203 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Mo El-Haj, Sultan Almujaiwel, Damith Premasiri, Tharindu Ranasinghe and Ruslan Mitkov. 2024. DARES: Dataset for Arabic Readability Estimation of School Materials. Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context @ LREC-COLING 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{elhaj2024dares,
  title = {{DARES: Dataset for Arabic Readability Estimation of School Materials}},
  author = {Mo El-Haj, Sultan Almujaiwel, Damith Premasiri, Tharindu Ranasinghe and Ruslan Mitkov},
  booktitle = {Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/4d3bc8zk39hb},
  url = {https://doi.org/10.63317/4d3bc8zk39hb}
}

References

45 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhamed Al Khalil, Hind Saddiki, Nizar Habash, and Latifa Alfalasi. 2018. A leveled reading corpus of Modern Standard Arabic. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  2. Sihaam Alotaibi, Maha Alyahya, Hend Al-Khalifa, Sinaa Alageel, and Nora Abanmy. 2016. Readability of arabic medicine information leaflets: a machine learning approach. Procedia Computer Science, 82:122–126. find
  3. Sandra Aluisio, Lucia Specia, Caroline Gasperin, and Carolina Scarton. 2010. Readability assessment for text simplification. In Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–9, Los Angeles, California. Association for Computational Linguistics.
  4. Fernando Alva-Manchego and Matthew Shardlow. 2022. Towards readability-controlled machine translation of COVID-19 texts. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 287–288, Ghent, Belgium. European Association for Machine Translation. find
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  7. Ibtehal Baazeem, Hend Al-Khalifa, and Abdulmalik Al-Salman. 2021. Cognitively driven arabic text readability assessment using eye-tracking. Applied Sciences, 11(18):8607. ↗
  8. Marc Benzahra and François Yvon. 2019. Measuring text readability with machine comprehension: a pilot study. In Workshop on Building Educational Applications Using NLP, pages 412–422, Florence, Italy. find
  9. Safae Berrichi, Naoual Nassiri, Azzeddine Mazroui, and Abdelhak Lakhouaja. 2022. Impact of feature vectorization methods on arabic text readability assessment. In The International Conference on Artificial Intelligence and Smart Environment, pages 504–510. Springer. find
  10. Sadik Bessou and Ghozlane Chenni. 2021. Efficient measuring of readability to improve documents accessibility for arabic language learners. ↗
  11. Violetta Cavalli-Sforza, Hind Saddiki, and Naoual Nassiri. 2018. Arabic readability research: Current state and future directions. Procedia Computer Science, 142:38–49. find
  12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics. ↗
  13. Scott Crossley, Aron Heintz, Joon Suh Choi, Jordan Batchelor, Mehrnoush Karimi, and Agnes Malatinszky. 2023. A large-scaled corpus for assessing text readability. Behavior Research Methods, 55(2):491–507. ↗
  14. Scott A Crossley, David B Allen, and Danielle S Mc-Namara. 2011. Text readability and intuitive simplification: A comparison of readability formulas. Reading in a foreign language, 23(1):84–101. ↗
  15. Edgar Dale and Jeanne S Chall. 1949. The concept of readability. Elementary English, 26(1):19–26. find
  16. Michael Dalvean and Galbadrakh Enkhbayar. 2018. Assessing the readability of fiction: A corpus analysis and readability ranking of 200 english fiction texts. Linguistic Research, 35:137–170. ↗
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  18. Mahmoud El-Haj, Abdulaziz Malik, and Michael K Paasche-Orlow. 2018. Readability of arabic vs english patient educational materials. In 2018 SGIM Annual Meeting. find
  19. Mahmoud El-Haj and Paul Edward Rayson. 2016. Osman: A novel arabic readability metric. In Language Resources and Evaluation (LREC). European Language Resources Association (ELRA). find
  20. Rudolph Flesch. 1948. A new readability yardstick. Journal of applied psychology, 32(3):221. ↗
  21. Thomas François and Eleni Miltsakaki. 2012. Do nlp and machine learning improve traditional readability formulas? In Proceedings of the First Workshop on Predicting and Improving Text Readability for target reader populations, pages 49–57. find
  22. Thomas François. 2015. When readability meets computational linguistics: a new paradigm in readability. Revue française de linguistique appliquée, 2:79–97. ↗
  23. Reem Hazim, Hind Saddiki, Bashar Alhafni, Muhamed Al Khalil, and Nizar Habash. 2023. Arabic word-level readability visualization for assisted text simplification. Computational Approaches to Modeling Language Lab.
  24. Joseph Marvin Imperial. 2021. BERT embeddings for automatic readability assessment. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 611–618, Held Online. IN-COMA Ltd. ↗
  25. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  26. Nouran Khallaf and Serge Sharoff. 2021. Automatic difficulty classification of Arabic sentences. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 105–114, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  27. Sidney Evaldo Leal, Magali Sanches Duran, and Sandra Aluísio. 2018. A nontrivial sentence corpus for the task of sentence readability assessment in portuguese. In Proceedings of the 27th International Conference on Computational Linguistics, pages 401–413. find
  28. Abdulaziz Malik, Mahmoud El-Haj, and Michael K Paasche-Orlow. 2019. Readability of patient educational materials in english versus arabic. HLRP: Health Literacy Research and Practice, 3(3):e170–e173. ↗
  29. Matej Martinc, Senja Pollak, and Marko Robnik-Šikonja. 2021. Supervised and unsupervised neural approaches to text readability. Computational Linguistics, 47(1):141–179. ↗
  30. Jorge Morato, Ana Iglesias, Adrián Campillo, and Sonia Sanchez-Cuadrado. 2021. Automated readability assessment for spanish egovernment information. Journal of Information Systems Engineering and Management, 6(2):em0137. find
  31. Babak Naderi, Salar Mohtaj, Karan Karan, and Sebastian Möller. 2019. Automated text readability assessment for german language: a quality of experience approach. In 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), pages 1–3. IEEE. find
  32. Naoual Nassiri, Violetta Cavalli-Sforza, and Abdelhak Lakhouaja. 2023. Approaches, methods, and resources for assessing the readability of arabic texts. ACM Transactions on Asian Low-Resource Language Information Processing (TALLIP), 22(4):95. ↗
  33. Kai North, Tharindu Ranasinghe, Matthew Shardlow, and Marcos Zampieri. 2023. Deep learning approaches to lexical simplification: A survey. arXiv preprint arXiv:2305.12000. ↗
  34. Kai North, Tharindu Ranasinghe, Matthew Shardlow, and Marcos Zampieri. 2024. Multils: A multi-task lexical simplification framework. arXiv preprint arXiv:2402.14972. ↗
  35. Kai North, Marcos Zampieri, and Tharindu Ranasinghe. 2022. ALEXSIS-PT: A new resource for Portuguese lexical simplification. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6057–6062, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. ↗
  36. Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In Proceedings of the 2008 conference on empirical methods in natural language processing, pages 186–195.
  37. Damith Premasiri, Tharindu Ranasinghe, Wajdi Zaghouani, and Ruslan Mitkov. 2022. DTW at qur’an QA 2022: Utilising transfer learning with transformers for question answering in a low-resource domain. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 88–95, Marseille, France. European Language Resources Association. ↗
  38. Xinying Qiu, Yuan Chen, Hanwu Chen, Jian-Yun Nie, Yuming Shen, and Dawei Lu. 2021. Learning syntactic dense embedding with correlation graph for automatic readability assessment. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3013–3025, Online. Association for Computational Linguistics. ↗
  39. Hind Saddiki, Karim Bouzoubaa, and Violetta Cavalli-Sforza. 2015. Text readability for arabic as a foreign language. In 2015 IEEE/ACS 12th International Conference of Computer Systems and Applications (AICCSA), pages 1–8. IEEE. find
  40. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  41. Kumiko Tanaka-Ishii, Satoshi Tezuka, and Hiroshi Terada. 2010. Sorting texts by readability. Computational Linguistics, 36(2). ↗
  42. Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment research. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5366–5377”, Marseille, France. European Language Resources Association. find
  43. Sowmya Vajjala and Ivana Lučić. 2018. Onestopenglish corpus: A new corpus for automatic readability assessment and text simplification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 297–304, New Orleans, Louisiana, USA. Association for Computational Linguistics.
  44. Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2016. Text readability assessment for second language learners. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 12–22.
  45. Mostafa Zamanian and Pooneh Heydari. 2012. Readability of texts: State of the art. Theory & Practice in Language Studies, 2(1). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/haj-2024-dares-dataset-arabic-readability-estimation/