Elamur Research⌕
☰
home / papers / 2026 / Can LLMs Control Readability? A Multi-Dimensional Evaluation Framework for CEFR-Controlled Arabic Generation

Can LLMs Control Readability? A Multi-Dimensional Evaluation Framework for CEFR-Controlled Arabic Generation

Paper · 2026 · Nour Rabih, Chatrine Qwaider and Ted Briscoe

Summary

Large language models can produce fluent Arabic, but it is unclear whether they can hit a requested proficiency level reliably. This matters for adaptive language-learning tools. The authors built an evaluation framework that checks CEFR-targeted Arabic generation from several angles rather than with one readability score. They combined CEFR-labelled learner and synthetic essays from ZAEBUC and ARWI and scored them with the BAREC model on the 19-level Taha-19 scale. This gave an empirical mapping between CEFR and Taha-19, along with lexical and syntactic reference profiles for each level. GPT-4o then wrote essays under several prompt designs. Some prompts named only the level. Others added profile-based linguistic targets, such as length, sentence length, POS and dependency counts, and parse-tree depth. Others added SAMER-derived keyword lists. The outputs were checked against the profiles and BAREC predictions. Structured prompting clearly improved alignment, and the CEFR-guided prompt with lexical constraints performed best. Prompts without such guidance gave only weak control. The work offers a first empirical basis for readability-aware Arabic generation in educational systems.

Key points

  • Data: 214 corrected ZAEBUC essays and 3,220 synthetic ARWI essays, with 161 leveled ARWI prompts (50 Beginner, 56 Intermediate, 55 Advanced). Beginner, Intermediate and Advanced were mapped to A1–A2, B1–B2 and C1–C2; 6 Unassessable essays were dropped.
  • Readability was predicted sentence by sentence with the BAREC model on the Taha-19 scale and averaged per document. A1–A2 essays clustered around Taha levels 6–10, B1–B2 around 10–13, and C1–C2 higher.
  • The SAMER lexicon (26,578 lemmas, 5 levels) was mapped heuristically to CEFR bands and used to supply level-appropriate vocabulary constraints.
  • The best setup, CEFR-guided prompting with lexical constraints, reached 0.91 cosine similarity to the reference linguistic profiles and 0.99 agreement with predicted readability levels.
  • The authors release a dataset of CEFR-controlled Arabic essays generated by GPT-4o on GitHub (noorrabih/CEFR-Controlled-Arabic-Generation-Data).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Nour Rabih, Chatrine Qwaider and Ted Briscoe. 2026. Can LLMs Control Readability? A Multi-Dimensional Evaluation Framework for CEFR-Controlled Arabic Generation. Proceedings of the Joint Workshop on Readability and Text Simplification (READIxTSAR) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/48v7mxywgfja

@inproceedings{rabih2026can,
  title = {{Can LLMs Control Readability? A Multi-Dimensional Evaluation Framework for CEFR-Controlled Arabic Generation}},
  author = {Nour Rabih, Chatrine Qwaider and Ted Briscoe},
  booktitle = {Proceedings of the Joint Workshop on Readability and Text Simplification (READIxTSAR) @ LREC 2026},
  year = {2026},
  doi = {10.63317/48v7mxywgfja},
  url = {https://doi.org/10.63317/48v7mxywgfja}
}

References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lexicon for Standard Arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062, Marseille, France. European Language Resources Association.
  2. Violetta Cavalli-Sforza, Hind Saddiki, and Naoual Nassiri. 2018. Arabic readability research: Current state and future directions. Procedia Computer Science, 142:38–49. Arabic Computational Linguistics. find
  3. C. o. E. Council of Europe. 2001. Common european framework of reference for languages: learning, teaching, assessment. find
  4. Mo El-Haj, Sultan Almujaiwel, Damith Premasiri, Tharindu Ranasinghe, and Ruslan Mitkov. 2024. Dares: Dataset for arabic readability estimation of school materials. In Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context@ LREC-COLING 2024, pages 103–113.
  5. Khalid N. Elmadani, Nizar Habash, and Hanada Taha-Thomure. 2025. A large and balanced corpus for fine-grained Arabic readability assessment. In Findings of the Association for Computational Linguistics: ACL 2025, pages 16376– 16400, Vienna, Austria. Association for Computational Linguistics.
  6. Ahmed Elshabrawy, Muhammed AbuOdeh, Go Inoue, and Nizar Habash. 2023. CamelParser2.0: A State-of-the-Art Dependency Parser for Arabic. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).
  7. Nizar Habash and David Palfreyman. 2022. Zaebuc: An annotated arabic-english bilingual writer corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79–88. find
  8. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274. ↗
  9. Nouran Khallaf and Serge Sharoff. 2021. Automatic difficulty classification of arabic sentences. arXiv preprint arXiv:2103.04386. ↗
  10. Tarek Naous, Michael J Ryan, Anton Lavrouk, Mohit Chandra, and Wei Xu. 2024. Readme++: Benchmarking multilingual language models for multi-domain readability assessment. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2024, page 12230. find
  11. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association.
  12. Chatrine Qwaider, Bashar Alhafni, Kirill Chirkunov, Nizar Habash, and Ted Briscoe. 2025. Enhancing arabic automated essay scoring with synthetic data and error injection. arXiv preprint arXiv:2503.17739. ↗
  13. Nour Rabih. 2025. Noor at barec shared task 2025: A hybrid transformer-feature architecture for sentence-level readability assessment. In Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks, pages 331–342. ↗
  14. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  15. Leonardo FR Ribeiro, Mohit Bansal, and Markus Dreyer. 2023. Generating summaries with controllable readability levels. arXiv preprint arXiv:2310.10623. Hanada Arabic ing Taha-Thomure. ↗
  16. Language Text 2017. Level- ñJ‚Ó J ’JË é£ @XA Jë Q K AªÓ ( ñ’ JË@ HAK éJ  K. QªË@ ). Educational    Q ( ©K Pñ JË@ð Qå „ JÊË ø ñK. Ë@ H. AJºË@ P@X ) . Book House Sean Trott and Pamela Rivière. 2024. Measuring and modifying the readability of english texts with gpt-4. In Proceedings of the Third Workshop on Text Simplification, Accessibility and Readability (TSAR 2024), pages 126–134.
  17. Jeromie Whalen, Chrystalla Mouza, et al. 2023. Chatgpt: Challenges, opportunities, and implications for teacher education. Contemporary Issues in Technology and Teacher Education, 23(1):1–23. find
  18. Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. 2016. Text readability assessment for second language learners. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 12–22, San Diego, CA. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rabih-2026-can-llms-control-readability-multi/