Elamur Research⌕
☰
home / papers / 2018 / Feature Optimization for Predicting Readability of Arabic L1 and L2

Feature Optimization for Predicting Readability of Arabic L1 and L2

Paper · 2018 · Hind Saddiki, Nizar Habash, Violetta Cavalli-Sforza et al.

Summary

The paper addresses automatic readability assessment for Arabic, a morphologically rich language with relatively few resources. Earlier Arabic studies looked at either native-speaker (L1) or learner (L2) texts. Here both are handled in one experimental setting: each document is assigned to one of four difficulty levels. The authors built a large L1 collection (27,688 documents) and a much smaller L2 one (576 documents). They then designed 146 document-level features organized along two axes. The first is how a feature is quantified: simple counts and ratios (Base) or n-gram perplexities (LM). The second is how much processing it needs: raw text, morphological analysis, or syntactic parsing with CATiB dependencies. Several classifiers were compared, and feature subsets were tuned with SVM. The full combined feature set gave the best results, 94.8% accuracy for L1 and 72.4% for L2. Adding language models trained on the large L1 data raised L2 accuracy to 74.1%. The work matters because it shows that deeper linguistic layers and resource transfer help in low-resource readability prediction.

Key points

  • Data scale: L1 has about 6.9M tokens in 27,688 documents, and L2 has 186,125 tokens in 576 documents. The authors describe this as the largest Arabic readability study to date and the first to treat L1 and L2 together.
  • There are 146 features. Raw Base features include the Al-Heeti, ARI and AARI formulas. Morph features include lemmas, morphemes, POS and verb aspect/voice. Syn features include CATiB dependency counts and parse tree depth and breadth. LM features are uni-, bi- and tri-gram perplexities, with a separate model built for each readability level.
  • Against the raw shallow baseline, the best feature set cuts errors by 75% for L1 (94.8%) and by 45% for L2 (72.4%). Parsing-based features alone reduce errors by 16% for L1 and 34% for L2.
  • L1-trained LM features push L2 accuracy to 74.1%, an extra 6% error reduction. Random Forest benefits most from them (45% accuracy error reduction), while SVM and Decision Tree change little.
  • Correlation-based feature selection produces compact subsets: 10 features for L1 and 29 for L2.

Interesting ideas

  • I1. Train a separate n-gram model for each difficulty level and use a text's perplexity under every model as features; this turns the question "which level does this text resemble?" into graded, inspectable numbers.
  • I2. Arrange features on a grid of quantification type versus processing depth, then run ablations cell by cell to see how much each deeper layer adds over a shallow baseline.
  • I3. Compute perplexity over sequences of POS tags and dependency labels, not only words, so that structural unusualness becomes a scalar signal.
  • I4. Use language models built on a large, rich resource as extra features for a small, sparse task; this helped the weaker L2 classifiers without hurting any of them.
  • I5. Shrink a large feature set to a small correlation-selected subset that keeps much of the gain, which makes the model easier to explain.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-course readers could estimate the grade level of a fable or test passage this way, so tasks are matched to the student's level with a visible reason.
  • I2 (learn): The same grid could measure whether tree- or frame-level signals actually beat surface cues, serving as a principled negative control against hidden bag-of-words shortcuts.
  • I3 (tree): A dependency-label LM trained on treebanks could flag improbable parses from the perceptron parser or gparse, and send them to repair rules or to a witness vote.
  • I4 (learn): Models trained on abundant silver data could feed features into small hand-checked tasks, such as dialect conversion or rare constructions, where gold data is scarce.
  • I5 (answer): A short list of named difficulty predictors would let a reader justify why it rated a text as hard, consistent with the explainable-at-every-step design.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations🎓Al Akhawayn UniversityIfrane · Morocco3 works · 247 citations🎓Mohammed V UniversityRabat · Morocco8 works · 103 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Hind Saddiki, Nizar Habash, Violetta Cavalli-Sforza and Muhamed Al Khalil. 2018. Feature Optimization for Predicting Readability of Arabic L1 and L2. Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-3703

@inproceedings{saddiki2018feature,
  title = {{Feature Optimization for Predicting Readability of Arabic L1 and L2}},
  author = {Hind Saddiki, Nizar Habash, Violetta Cavalli-Sforza and Muhamed {Al Khalil}},
  booktitle = {Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications},
  year = {2018},
  doi = {10.18653/v1/w18-3703},
  url = {https://doi.org/10.18653/v1/w18-3703}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hend S Al-Khalifa and Amani A Al-Ajlan. 2010. Automatic readability measurements of the Arabic text: An exploratory study. Arabian Journal for Science and Engineering, 35(2 C):103–124. find
  2. Lijun Feng, Noémie Elhadad, and Matt Huenerfauth. 2009. Cognitively motivated features for readability assessment. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, pages 229–237. Association for Computational Linguistics. find
  3. Muhamed Al Khalil, Nizar Habash, and Hind Saddiki. 2017. Simplification of Arabic masterpieces for extensive reading: A project overview. Procedia Computer Science, 117:192–198. find
  4. Jonathan Forsyth. 2014. Automatic readability prediction for modern standard Arabic. In Proceedings of the First Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools (LREC 2014), Reykjavik, Iceland. find
  5. Abdel Karim Al Tamimi, Manar Jaradat, Nuha Al-Jarrah, and Sahar Ghanem. 2014. AARI: automatic Arabic readability index. Int. Arab J. Inf. Technol., 11(4):370–378. find
  6. Sandra Aluisio, Lucia Specia, Caroline Gasperin, and Carolina Scarton. 2010. Readability assessment for text simplification. In Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–9. Association for Computational Linguistics.
  7. Thomas François and Eleni Miltsakaki. 2012. Do nlp and machine learning improve traditional readability formulas? In Proceedings of the First Workshop on Predicting and Improving Text Readability for target reader populations, pages 49–57. Association for Computational Linguistics. find
  8. Lisa Beinborn, Torsten Zesch, and Iryna Gurevych. 2012. Towards fine-grained readability measures for self-directed language learning. In Proceedings of the SLTC 2012 workshop on NLP for CALL; Lund; N. Habash, O. Rambow, and R. Roth. 2012. MADA+ TOKAN Manual. Technical report, Technical Report CCLS-12-01, Columbia University. find
  9. Hind Saddiki, Karim Bouzoubaa, and Violetta Cavalli-Sforza. 2015. Text readability for Arabic as a foreign language. In Proceedings of the IEEE/ACS 12th International Conference of Computer Systems and Applications (AICCSA), Marrakech, Morocco, pages 1–8. IEEE. find
  10. Nizar Habash and Ryan M Roth. 2009. CATiB: The Columbia Arabic Treebank. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 221–224. Association for Computational Linguistics. ↗
  11. Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. find
  12. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. 2009. The weka data mining software: an update. ACM SIGKDD explorations newsletter, 11(1):10– 18. ↗
  13. Mark Andrew Hall. 1999. Correlation-based feature selection for machine learning. Ph.D. thesis, University of Waikato Hamilton. find
  14. Julia Hancke, Sowmya Vajjala, and Detmar Meurers. 2012. Readability classification for german using lexical, syntactic, and morphological features. In COLING, pages 1063–1080. find
  15. Haibo He and Edwardo A Garcia. 2009. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9):1263–1284. ↗
  16. Michael Heilman, Kevyn Collins-Thompson, Jamie Callan, and Maxine Eskenazi. 2007. Combining lexical and grammatical features to improve readability measures for first and second language texts. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 460–467. find
  17. Michael Heilman, Kevyn Collins-Thompson, and Maxine Eskenazi. 2008. An analysis of statistical models and features for reading difficulty prediction. In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications, pages 71–79. Association for Computational Linguistics. find
  18. Muhamed Al Khalil, Hind Saddiki, Nizar Habash, and Latifa Alfalasi. 2018. A Leveled Reading Corpus of Modern Standard Arabic. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
  19. Yuval Marton, Nizar Habash, and Owen Rambow. 2013. Dependency parsing of modern standard Arabic with lexical and inflectional features. Computational Linguistics, 39(1):161–194. ↗
  20. Naoual Nassiri, Abdelhak Lakhouaja, and Violetta Cavalli-Sforza. 2017. Modern standard Arabic readability prediction. In International Conference on Arabic Language Processing, pages 120–133. Springer. find
  21. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan M Roth. 2014. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland.
  22. Anas Shahrour, Salam Khalifa, Dima Taji, and Nizar Habash. 2016. Camelparser: A system for arabic syntactic analysis and morphological disambiguation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pages 228–232.
  23. Andreas Stolcke et al. 2002. Srilm-an extensible language modeling toolkit. In Interspeech, volume 2002, page 2002. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saddiki-2018-feature-optimization-predicting-readability-arabic/