Elamur Research⌕
☰
home / papers / 2026 / Transformer-based readability classifiers are worse than you think: Evidence from cross-domain Arabic readability assessment

Transformer-based readability classifiers are worse than you think: Evidence from cross-domain Arabic readability assessment

Paper · 2026 · Sarah Alzubi and Robert Reynolds

Summary

Most Arabic readability models are tested only on data resembling their training texts, so it is unclear whether they would hold up on other kinds of text. The authors create a new resource drawn from 240 national-curriculum textbooks covering grades 1–12, which they call the Jordanian School Textbook Corpus (JSTC); the books were digitized with QARI-OCR and cleaned with CAMeL Tools. They merge it with the Saudi DARES corpus to form a training set. On this data they compare three model families: XGBoost over 127 handcrafted linguistic features, fine-tuned CAMeLBERT, and hybrids that feed [CLS] embeddings alongside the features. Every model is scored in-domain and also on BAREC, a multi-domain benchmark excluded from training. CAMeLBERT looks excellent in-domain but drops almost to chance on BAREC. The feature-only XGBoost transfers best, and adding embeddings to the features makes transfer worse. Probing and attribution analyses examine what the transformer actually encodes. The study argues that cross-domain testing should be routine and that interpretable linguistic features are the safer option when the incoming texts will vary.

Key points

  • JSTC contains 82,512 segments (over 2.1M tokens) from 240 Jordanian textbooks. Deduplication removed 5.42% of JSTC and 19.37% of DARES V1, leaving about 93,265 combined training segments.
  • The 127 features fall into six groups. Examples include TTR/MATTR/MTLD, lexical density, TF-IDF statistics over surface and lemma forms, and POS distributions. Syntactic features such as clause depth and dependency structure come from CamelParser 2.0 with Universal Dependencies.
  • CAMeLBERT reaches QWK 0.830 in-domain but only 0.085 on BAREC. XGBoost with linguistic features alone gets the best cross-domain QWK, 0.240–0.264.
  • Hybrid models that combine features with [CLS] embeddings transfer worse than features alone. This contradicts earlier English findings that hybrids win (Lee et al., 2021).
  • Probing shows that CAMeLBERT layers partly encode linguistic features, but other signals dominate its decisions. Captum attribution ranks nouns and nominal particles like al- as the most influential tokens.

Interesting ideas

  • I1. Keep one benchmark entirely out of training and report it next to the in-domain score; a high in-domain number alone can hide a near-total collapse.
  • I2. Adding a strong neural representation to good features can make generalization worse, so each fusion should be checked by ablation under shift rather than assumed to help.
  • I3. Clause embedding depth, dependency structure and lexical diversity, all computed from a UD parse, serve as interpretable difficulty signals that survive domain change.
  • I4. Layer-wise probing for linguistic properties, combined with token attribution, reveals what a fine-tuned model actually relies on.
  • I5. The authors admit that their test differs in both domain and format (single sentences vs. passages), and suggest rotating which corpus is held out to separate these effects.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Every trained component, such as the perceptron parser or gparse, could carry a mandatory out-of-genre test set, so regressions under shift show up before deployment.
  • I2 (tree): When mixing transformer signals from gparse with explicit features, compare the feature-only, neural-only and combined versions on out-of-domain text before keeping the combination.
  • I3 (answer): School-course readers could estimate the grade level of a text or question from its UD tree, and explain the estimate through named structural features.
  • I4 (learn): Probing gparse layers for morphology and syntax, and attributing its decisions to tokens, would help verify that it learns structure rather than corpus quirks.
  • I5 (learn): Evaluation design could change one factor at a time, such as genre or unit length, and rotate held-out treebanks, so that failures can be attributed to a specific cause.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Brigham Young UniversityProvo · United States16 works · 40,099 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Sarah Alzubi and Robert Reynolds. 2026. Transformer-based readability classifiers are worse than you think: Evidence from cross-domain Arabic readability assessment. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.52

@inproceedings{alzubi2026transformer,
  title = {{Transformer-based readability classifiers are worse than you think: Evidence from cross-domain Arabic readability assessment}},
  author = {Sarah Alzubi and Robert Reynolds},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.52},
  url = {https://doi.org/10.18653/v1/2026.bea-1.52}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdel-Karim Al-Tamimi, Manar Jaradat, Nuha Aljarrah, and Sahar Ghanim. 2014. AARI: Automatic Arabic readability index. Information Technology Journal, 13:1350–1357. find
  2. Sultan Almujaiwel, Damith Premasiri, Tharindu Ranasinghe, Mo El-Haj, and Ruslan Mitkov. 2025. Complex concept-based readability estimation from Arabic curriculum. ACM Transactions on Asian and Low-Resource Language Information Processing, 24(11):1–21. ↗
  3. Sarah Alzubi. 2026. Robust and interpretable cross-domain arabic readability prediction using hybrid modeling: Evidence on the limitations of transformer-based classifiers. Master’s thesis, Brigham Young University, April. find
  4. Mohammed Attia, Younes Samih, and Yo Ehara. 2023. Statistical measures for readability assessment. In Proceedings of the Joint 3rd International Conference on Natural Language Processing for Digital Humanities and 8th International Workshop on Computational Linguistics for Uralic Languages, pages 153–161, Tokyo, Japan. Association for Computational Linguistics. find
  5. Violetta Cavalli-Sforza, Hind Saddiki, and Naoual Nassiri. 2018. Arabic readability research: Current state and future directions. Procedia Computer Science, 142:38–49. find
  6. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794. ACM. ↗
  7. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213–220. ↗
  8. Michael A. Covington and Joe D. McFall. 2010. Cutting the Gordian knot: The moving-average type–token ratio (MATTR). Journal of Quantitative Linguistics, 17(2):94–100. ↗
  9. Tovly Deutsch, Masoud Jasbi, and Stuart Shieber. 2020. Linguistic features for readability assessment. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–17. Association for Computational Linguistics.
  10. Mahmoud El-Haj and Paul Rayson. 2016. OSMAN – a novel Arabic readability metric. Proceedings of LREC 2016. find
  11. Mo El-Haj, Sultan Almujaiwel, Damith Premasiri, Tharindu Ranasinghe, and Ruslan Mitkov. 2024. DARES: Dataset for Arabic readability estimation of school materials. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association.
  12. Khalid N. Elmadani, Nizar Habash, and Hanada Taha-Thomure. 2025. A large and balanced corpus for fine-grained Arabic readability assessment. arXiv preprint.
  13. Ahmed Elshabrawy, Muhammed AbuOdeh, Go Inoue, and Nizar Habash. 2023. CamelParser2.0: A state-ofthe-art dependency parser for Arabic. In Proceedings of ArabicNLP 2023, pages 170–180. Association for Computational Linguistics. find
  14. Jonathan Neil Forsyth. 2014. Automatic readability detection for Modern Standard Arabic. Master’s thesis, Brigham Young University. find
  15. Joseph Marvin Imperial. 2021. BERT embeddings for automatic readability assessment. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 611–618. INCOMA Ltd. ↗
  16. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104. Association for Computational Linguistics.
  17. Nouran Khallaf and Serge Sharoff. 2021. Automatic difficulty classification of Arabic sentences. arXiv preprint. ↗
  18. Bruce W. Lee, Yoo Sung Jang, and Jason Lee. 2021. Pushing on text readability assessment: A transformer meets handcrafted linguistic features. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10669– 10686. Association for Computational Linguistics.
  19. Ho Hung Lim, Tianyuan Cai, John S. Y. Lee, and Meichun Liu. 2024. Robustness of hybrid models in cross-domain readability assessment. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. find
  20. Philip M. McCarthy and Scott Jarvis. 2010. MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2):381–392. ↗
  21. Tarek Naous, Michael J. Ryan, Anton Lavrouk, Mohit Chandra, and Wei Xu. 2024. ReadMe++: Benchmarking multilingual language models for multi-domain readability assessment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
  22. Naoual Nassiri, Violetta Cavalli-Sforza, and Abdelhak Lakhouaja. 2019. MoSAR: Modern Standard Arabic readability corpus for L1 learners. In Proceedings of the 4th International Conference on Big Data and Internet of Things, pages 1–7. ACM. find
  23. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source Python toolkit for Arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032. European Language Resources Association.
  24. Mohamed Amine Ouassil, Mohammed Jebbari, Rabia Rachidi, Mouaad Errami, Bouchaib Cherradi, and Abdelhadi Raihani. 2024. Enhancing Arabic text readability assessment: A combined BERT and BiL-STM approach. In 2024 International Conference on Circuit, Systems and Communication (ICCSC), pages 1–7. IEEE. find
  25. Sowmya Vajjala. 2022. Trends, limitations and open challenges in automatic readability assessment research. arXiv preprint arXiv:2105.00973. ↗
  26. Frans Van Der Sluis and Egon L. Van Den Broek. 2025. Model interpretability enhances domain generalization in the case of textual complexity modeling. Patterns, 6(2):101177. ↗
  27. Ahmed Wasfy, Omer Nacar, Abdelakreem Elkhateb, Mahmoud Reda, Omar Elshehy, Adel Ammar, and Wadii Boulila. 2025. QARI-OCR: High-fidelity Arabic text recognition through multimodal large language model adaptation. arXiv preprint arXiv:2506.02295. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alzubi-2026-transformer-based-readability-classifiers-are/