Elamur Research⌕
☰
home / papers / 2024 / Strategies for Arabic Readability Modeling

Strategies for Arabic Readability Modeling

Paper · 2024 · Juan Liberato, Bashar Alhafni, Muhamed Khalil et al.

Summary

The paper tackles automatic readability assessment for Modern Standard Arabic, which is hard because of rich morphology, orthographic ambiguity, and few resources. The authors work at two granularities: single words and short sentence fragments. They use the SAMER Arabic Text Simplification Corpus, in which fiction excerpts were manually simplified to two easier levels. From these parallel versions they automatically derive a label of Level 3, 4 or 5 for every original word, and a fragment's label is its hardest word. They compare several approaches: a frequency-based MLE model built from corpus counts, a lookup in the SAMER Readability Lexicon guided by a morphological disambiguator, and a fine-tuned Arabic BERT classifier. They also test cascades in which each model backs off to the next. The best cascade, Tuned-MLE → Lex → BERT, reaches 86.7 macro F1 on words and 87.9 on fragments on a blind test set, well above BERT alone. The work matters because it shows that curated lexical resources and simple statistics still add a lot to pretrained models for a low-resource, morphologically complex language.

Key points

  • Data: the SAMER Corpus contains texts from 15 Arabic novels, each with Level 4 and Level 3 simplifications on a five-level lexical scale. It has 20,358 fragments (14,256 train / 2,948 dev / 3,154 test) and 158,054 word tokens. The tokens are highly skewed: 86.6% L3, 8.9% L4, 4.5% L5.
  • Word labels come from an edit-distance word alignment between the original and simplified versions. A word gets the lowest level at which it still survives unchanged.
  • The Lex model maps each word to its lemma using the disambiguator of Inoue et al. (2022) in CAMeL Tools. It then looks the lemma up in the SAMER Readability Lexicon (40K+ lemmas), merging Levels 1–2 into Level 3. When several top analyses tie, it takes the lowest level.
  • On the test set, BERT alone scores 74.7 macro F1 on words and 75.5 on fragments, and Lexicon → BERT scores 85.0 and 85.4. The full cascade reaches 86.7 and 87.9. The MLE component made most decisions with the fewest errors, while BERT's decisions were wrong 35.6% of the time at word level and 67.7% at fragment level.
  • The authors manually analysed 100 of 814 dev-set word errors:
    • disambiguation 31%
    • MLE 22%
    • gold annotation 18%
    • lexicon gaps 11%
    • gold label-derivation 8%
    • BERT 7%
    • malformed input 3%

Interesting ideas

  • I1. Word-level labels are obtained for free by aligning an original text with its graded rewrites: whatever the editors left untouched at a level is treated as acceptable for that level. This turns any parallel simplification or paraphrase data into token supervision.
  • I2. The models are ordered as a back-off cascade: an exact-match statistic answers first, a curated lexicon second, and the neural model only handles what remains. The authors record which component made each decision and how often it erred.
  • I3. When analysis is ambiguous among top lemma candidates, the system deliberately picks the easiest level, a fixed and conservative tie-breaking policy instead of a random choice.
  • I4. A fragment's difficulty is the maximum of its words' levels, which makes the aggregate fully traceable to one culprit word. The authors measure how word mistakes inflate fragment errors (3.7% word errors leading to 11.4% fragment errors).
  • I5. The error taxonomy is applied in a strict precedence order and explicitly includes errors in the gold labels and in the label-derivation procedure, not just model errors.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): parallel easy/hard versions of school texts could yield silver word-difficulty or "safe-to-keep" labels without manual tagging, with the alignment step logged so each label stays explainable.
  • I2 (tokens): the lemma/morphology pipeline could use the same priority chain (seen-form table → WordNet lexicon → neural guess), reporting per-stage error rates so weak fallbacks are visible.
  • I3 (tokens): ties in lemma or POS analysis could be resolved by an explicit, documented preference rule, so the downstream tree and frames inherit a predictable choice.
  • I4 (answer): when grading or choosing texts for student readers, a max-over-words rule could point to the exact word that makes a passage too hard, and the error-propagation measure would show how token errors amplify.
  • I5 (learn): evaluation of silver data could classify failures in a fixed order (input → reference → derivation → component), separating real parser mistakes from faulty references.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Juan Liberato, Bashar Alhafni, Muhamed Khalil and Nizar Habash. 2024. Strategies for Arabic Readability Modeling. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.5

@inproceedings{liberato2024strategies,
  title = {{Strategies for Arabic Readability Modeling}},
  author = {Juan Liberato, Bashar Alhafni, Muhamed Khalil and Nizar Habash},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.5},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.5}
}

References

57 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Hend S Al-Khalifa and Amani A Al-Ajlan. 2010. Automatic readability measurements of the Arabic text: An exploratory study. Arabian Journal for Science and Engineering, 35(2 C):103–124. find
  3. Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020a. A Large-Scale Leveled Readability Lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.
  4. Miriam Cha, Youngjune Gwon, and H. T. Kung. 2017. Language modeling by clustering with word embeddings for text readability assessment. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM ’17, page 2003–2006, New York, NY, USA. Association for Computing Machinery. find
  5. Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020b. A large-scale leveled readability lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.
  6. Kevyn Collins-Thompson and James P. Callan. 2004a. A language modeling approach to predicting reading difficulty. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 193–200, Boston, Massachusetts, USA. Association for Computational Linguistics.
  7. Abdel Karim Al Tamimi, Manar Jaradat, Nuha Al-Jarrah, and Sahar Ghanem. 2014. AARI: automatic Arabic readability index. International Arab Journal of Information Technology, 11(4):370–378. find
  8. Kevyn Collins-Thompson and Jamie Callan. 2004b. Information retrieval for language tutoring: an overview of the reap project. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’04, page 544–545, New York, NY, USA. Association for Computing Machinery. find
  9. Abdel-Karim Al-Tamimi, Manar Jaradat, Nuha Aljarrah, and Sahar Ghanim. 2014. Aari: Automatic arabic readability index. International Arab Journal of Information Technology, 11:370–378. find
  10. Bashar Alhafni, Reem Hazim, Juan David Pineros Liberato, Muhamed Al Khalil, and Nizar Habash. 2024. The SAMER Arabic text simplification corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16079–16093, Torino, Italia. ELRA and ICCL.
  11. C. o. E. Council of Europe. 2001. Common european framework of reference for languages: learning, teaching, assessment. Cambridge University Press. find
  12. Edgar Dale and Jeanne S. Chall. 1948. A formula for predicting readability. Educational Research Bulletin, 27(1):11–28. find
  13. Bashar Alhafni, Go Inoue, Christian Khairallah, and Nizar Habash. 2023. Advancements in Arabic grammatical error detection and correction: An empirical investigation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6430–6448, Singapore. Association for Computational Linguistics.
  14. Tovly Deutsch, Masoud Jasbi, and Stuart Shieber. 2020. Linguistic features for readability assessment. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–17, Seattle, WA, USA → Online. Association for Computational Linguistics.
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  16. Bharat Ram Ambati, Siva Reddy, and Mark Steedman. 2016. Assessing relative sentence complexity using an incremental CCG parser. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1051–1057, San Diego, California. Association for Computational Linguistics. find
  17. Mahmoud El-Haj and Paul Rayson. 2016. OSMAN: A novel Arabic readability metric. In Proceedings of the Language Resources and Evaluation Conference (LREC), Portorož, Slovenia. find
  18. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  19. Lijun Feng, Martin Jansche, Matt Huenerfauth, and Noémie Elhadad. 2010. A comparison of features for automatic readability assessment. In Coling 2010: Posters, pages 276–284, Beijing, China. Coling 2010 Organizing Committee.
  20. Ion Madrazo Azpiazu and Maria Soledad Pera. 2019. Multiattentive recurrent neural network architecture for multilingual readability assessment. Transactions of the Association for Computational Linguistics, 7:421–436. ↗
  21. Rudolph Flesch. 1948. A new readability yardstick. Journal of Applied Psychology, 32(3):p221 – 233. ↗
  22. Jonathan Forsyth. 2014. Automatic readability prediction for modern standard Arabic. In Proceedings of the Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT). find
  23. Violetta Cavalli-Sforza, Mariam El Mezouar, and Hind Saddiki. 2014. Matching an Arabic text to a learners’ curriculum. In Proceedings of the Conference on Arabic Language Processing (CITALA), pages 79– 88, Oujda, Morocco. find
  24. Abbas Ghaddar, Yimeng Wu, Sunyam Bagga, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Xinyu Duan, Zhefeng Wang, Baoxing Huai, Xin Jiang, Qun Liu, and Phillippe Langlais. 2022. Revisiting pre-trained language models and their evaluation for Arabic natural language processing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3135–3151, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. knowledge for readability assessment. In Proceedings of the 27th International Conference on Computational Linguistics, pages 366–378, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  25. Salam Khalifa, Ossama Obeid, and Nizar Habash. 2021. Character Edit Distance Based Word Alignment. https://github.com/CAMeL-Lab/ ced_word_alignment. ↗
  26. Muhamed Al Khalil, Hind Saddiki, Nizar Habash, and Latifa Alfalasi. 2018. A Leveled Reading Corpus of Modern Standard Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan.
  27. Nouran Khallaf and Serge Sharoff. 2021. Automatic difficulty classification of Arabic sentences. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 105–114, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  28. Bruce W. Lee, Yoo Sung Jang, and Jason Lee. 2021. Pushing on text readability assessment: A transformer meets handcrafted linguistic features. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10669– 10686, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  29. Justin Lee and Sowmya Vajjala. 2022. A neural pairwise ranking model for readability assessment. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3802–3813, Dublin, Ireland. Association for Computational Linguistics. ↗
  30. Kelly Marchisio, Jialiang Guo, Cheng-I Lai, and Philipp Koehn. 2019. Controlling the reading level of machine translation output. In Proceedings of Machine Translation Summit XVII: Research Track, pages 193– 203, Dublin, Ireland. European Association for Machine Translation. find
  31. Matej Martinc, Senja Pollak, and Marko Robnik-Šikonja. 2021. Supervised and unsupervised neural approaches to text readability. Computational Linguistics, 47(1):141–179. ↗
  32. Farah Nadeem and Mari Ostendorf. 2018. Estimating linguistic complexity for science texts. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 45–55, New Orleans, Louisiana. Association for Computational Linguistics. find
  33. Tarek Naous, Michael J. Ryan, Anton Lavrouk, Mohit Chandra, and Wei Xu. 2023. Readme++: Benchmarking multilingual language models for multi-domain readability assessment. Preprint, arXiv:2305.14463. ↗
  34. Naoual Nassiri, Abdelhak Lakhouaja, and Violetta Cavalli-Sforza. 2017. Modern standard Arabic readability prediction. In Proceedings of the International Conference on Arabic Language, pages 120– 133. find
  35. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. find
  36. Nizar Habash and David Palfreyman. 2022. ZAEBUC: An annotated Arabic-English bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79–88, Marseille, France. European Language Resources Association. find
  37. Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. find
  38. Reem Hazim, Hind Saddiki, Bashar Alhafni, Muhamed Al Khalil, and Nizar Habash. 2022. Arabic word-level readability visualization for assisted text simplification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 242–249, Abu Dhabi, UAE. Association for Computational Linguistics.
  39. Michael Heilman, Kevyn Collins-Thompson, Jamie Callan, and Maxine Eskenazi. 2007. Combining lexical and grammatical features to improve readability measures for first and second language texts. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 460–467, Rochester, New York. Association for Computational Linguistics. find
  40. Joseph Marvin Imperial and Ekaterina Kochmar. 2023. Automatic readability assessment for closely related languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5371–5386, Toronto, Canada. Association for Computational Linguistics. ↗
  41. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  42. Go Inoue, Salam Khalifa, and Nizar Habash. 2022. Morphosyntactic tagging with pre-trained language models for Arabic and its dialects. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1708–1719, Dublin, Ireland. Association for Computational Linguistics.
  43. Zhengyang Jiang, Nizar Habash, and Muhamed Al Khalil. 2020. An online readability leveled Arabic thesaurus. In Proceedings of the 28th International Conference on Computational Linguistics: System Demonstrations, pages 59–63, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL).
  44. Zhiwei Jiang, Qing Gu, Yafeng Yin, and Daoxu Chen. 2018. Enriching word embeddings with domain CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association. find
  45. Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗
  46. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  47. Sarah E. Petersen and Mari Ostendorf. 2009. A machine learning approach to reading level assessment. Computer Speech & Language, 23(1):89–106. ↗
  48. Emily Pitler and Ani Nenkova. 2008. Revisiting readability: A unified framework for predicting text quality. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 186–195, Honolulu, Hawaii. Association for Computational Linguistics.
  49. Hind Saddiki, Karim Bouzoubaa, and Violetta Cavalli-Sforza. 2015. Text readability for Arabic as a foreign language. In Proceedings of the International Conference of Computer Systems and Applications (AICCSA), pages 1–8, Marrakech, Morocco. find
  50. Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2016. Text readability assessment for second language learners. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 12–22, San Diego, CA. Association for Computational Linguistics.
  51. Hind Saddiki, Nizar Habash, Violetta Cavalli-Sforza, and Muhamed Al Khalil. 2018. Feature Optimization for Predicting Readability of Arabic L1 and L2. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, pages 20–29, Melbourne, Australia. Association for Computational Linguistics. ↗
  52. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297. ↗
  53. Luo Si and Jamie Callan. 2001. A statistical model for scientific readability. In Proceedings of the Tenth International Conference on Information and Knowledge Management, CIKM ’01, page 574–576, New York, NY, USA. Association for Computing Machinery. find
  54. Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment research. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5366– 5377, Marseille, France. European Language Resources Association. find
  55. Sowmya Vajjala and Ivana Lučić. 2018. OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 297–304, New Orleans, Louisiana. Association for Computational Linguistics.
  56. Sowmya Vajjala and Detmar Meurers. 2012. On improving the accuracy of readability classification using insights from second language acquisition. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 163–173, Montréal, Canada. Association for Computational Linguistics.
  57. Sowmya Vajjala and Detmar Meurers. 2013. On the applicability of readability models to web texts. In Proceedings of the Second Workshop on Predicting and Improving Text Readability for Target Reader Populations, pages 59–68, Sofia, Bulgaria. Association for Computational Linguistics. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liberato-2024-strategies-arabic-readability-modeling/