Elamur Research⌕
☰
home / papers / 2021 / Efficient Measuring of Readability to Improve Documents Accessibility for Arabic Language Learners

Efficient Measuring of Readability to Improve Documents Accessibility for Arabic Language Learners

Summary

The paper tackles automatic difficulty grading for Arabic texts so that learners can be given material that fits their level. The authors argue that Arabic has received far less attention here than English. They built a corpus of 39,792 web-collected Arabic documents, each labelled with one of four levels. Labels follow the intended audience of each source: children's books and primary textbooks at the bottom, pre-Islamic classical writing at the top. Texts were cleaned, normalized for Hamza/Alif spelling variants, stripped of stop words and light-stemmed. The authors then compared Count and TF-IDF vectors built from unigrams, bigrams or both, feeding each into five standard classifiers. The strongest setup was an SVM on TF-IDF with combined unigrams and bigrams, at 87.14% accuracy over four classes. Multinomial Naïve Bayes on Count vectors came close. The work matters because it provides a sizeable labelled Arabic readability resource. It also shows that the best classifier depends on how features are represented.

Key points

  • The corpus has 39,792 documents and 475,454 tokens. They split into Easy (6,691), Medium (8,844), Difficult (9,326) and Very difficult (14,931).
  • The "Very difficult" class consists of pre-Islamic texts in a variety that differs from MSA. The top level therefore partly reflects a change of language variety, not only complexity.
  • Five classifiers were tested: Multinomial NB, Bernoulli NB, Logistic Regression, SVM and Random Forest. Each was run across 2 representations × 3 n-gram settings, giving six experiments in total.
  • The best result was SVM with TF-IDF on unigrams plus bigrams, at 87.14% accuracy and F1 0.87. With Count vectors, Multinomial NB led at 86.47%, while Bernoulli NB was consistently weakest (as low as 44.74%).
  • Bigrams on their own performed much worse (best 76.74%). Adding them to unigrams helped every classifier slightly, and SVM did better with TF-IDF while Multinomial NB did better with raw counts.

Interesting ideas

  • I1. Difficulty labels come cheaply from where a text was published (primary textbook, student magazine, university material), so no per-document human rating is needed; this provenance-as-label trick scales annotation for graded corpora. — For Mova (answer): school-course test passages could be tagged by the grade of their source book, letting us report reader accuracy per difficulty band instead of one flat score.
  • I2. A full grid of representation × n-gram setting × classifier showed that rankings flip between setups, e.g. NB wins on counts while SVM wins on TF-IDF, which guards against declaring a model best from a single configuration. — For Mova (learn): evaluations of parser or frame-extractor variants should cross feature/encoding choices with model types before crediting any one component.
  • I3. Sparse bigram features fail on their own yet add a little on top of unigrams, a reminder that higher-order features work as supplements rather than replacements. — For Mova (tokens): tagger feature templates could add word-pair context only alongside single-word features, with an ablation that keeps the bigram-only setting as a check.
  • I4. A shallow n-gram classifier already reaches high accuracy on level prediction, which makes it a natural baseline that deeper analyses must beat. — For Mova (learn): use such a bag-of-n-grams grader as a negative control, so that any difficulty signal derived from UD trees or frames has to show gains beyond surface lexis.
  • I5. Spelling variants (Hamza/Alif forms) are unified before features are counted, so that one word does not split into several sparse entries. — For Mova (tokens): map orthographic variants such as British/American spellings or quote and dash forms to one key before lemma lookup, while keeping the original form for explainability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages · Grammatical error correction & learner language

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2109.08648 · arXiv 2109.08648

@misc{bessou2021efficient,
  title = {{Efficient Measuring of Readability to Improve Documents Accessibility for Arabic Language Learners}},
  author = {Bessou, Sadik and Chenni, Ghozlane},
  year = {2021},
  doi = {10.48550/arxiv.2109.08648},
  eprint = {2109.08648},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2109.08648}
}

References

20 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zambrano, J. O., Tapia, E. V. (2018). Reading comprehension in university texts: the metrics of lexical complexity in corpus analysis in Spanish. In: International Conference on Computer and Communication Engineering (p. 111-123). Springer, Cham, (October). find
  2. Zhang, Lixiao, Liu, Zaiying., Ni, Jun. (2013). Feature-based assessment of text readability. In: 2013 Seventh International Conference on Internet Computing for Engineering and Science, p. 51-54. IEEE. find
  3. Silpa, K. S., Irshad, M. (2018). Lexical Simplification of Complex Scientific Terms. In: 2018 International Conference on Emerging Trends and Innovations In Engineering and Technological Research (ICETIETR), p. 1-5. IEEE. find
  4. Wibowo, Satrio, Muhammad., Romadhony, Ade., Siti Sa’adah. (2019). Lexical and Syntactic Simplification for Indonesian Text. In: 2019 International Seminar on Research of Information Technology and Intelligent Systems (ISRITI), p. 64-68. IEEE. find
  5. Nandhini, Kumaresh., Ramakrishnan Balasundaram, Sadhu. (2012). Significance of learner dependent features for improving text readability using extractive summarization. In: 2012 4th International Conference on Intelligent Human Computer Interaction (IHCI), p. 1-5. IEEE. find
  6. Daowadung, Patcharanut., Yaw-Huei Chen. (2012). Stop word in readability assessment of Thai text. In: 2012 IEEE 12 th International Conference on Advanced Learning Technologies, p. 497-499. IEEE. find
  7. Luong, An-Vinh, Nguyen, Diep., DienDinh. (2017). Examining the text-length factor in evaluating the readability of literary texts in Vietnamese textbooks. In: 2017 9th International Conference on Knowledge and Systems Engineering (KSE), p. 36-41. IEEE. find
  8. Luong, An-Vinh, Nguyen, Diep., DienDinh. (2018). Assessing the Readability of Literary Texts in Vietnamese Textbooks. In: 2018 5 th NAFOSTED Conference on Information and Computer Science (NICS), p. 231-236. IEEE. find
  9. Mukherjee, Partha, Leroy, Gondy., Kauchak, David. (2018). Using Lexical Chains to Identify Text Difficulty: A Corpus Statistics and Classification Study. IEEE journal of biomedical and health informatics, 23 (5) 2164-2173. find
  10. Naderi, Babak, SalarMohtaj, Karan, Karan., Möller, Sebastian. (2019). Automated Text Readability Assessment for German Language: A Quality of Experience Approach. In: 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), p. 1-3. IEEE. find
  11. Narasinh, Vishwaas. (2019). Readability Analysis of Kannada Language. In: 2019 1 st International Conference on Advances in Information Technology (ICAIT), p. 45-49. IEEE. find
  12. Zhang, Yusha, Lin, Nankai., Jiang, Shengyi. (2019). A Study on Syntactic Complexity and Text Readability of ASEAN English News. In: 2019 International Conference on Asian Language Processing (IALP), p. 313-318. IEEE. find
  13. Al-Ajlan, Amani, A., Hend, S., Al-Khalifa, Abdul Malik S., Al-Salman. (2008). Towards the development of an automatic readability measurements for Arabic language. In: 2008 Third International Conference on Digital Information Management, p. 506-511. IEEE. find
  14. Saddiki, Hind, Bouzoubaa, Karim., Cavalli-Sforza, Violetta. (2015). Text readability for Arabic as a foreign language. In: 2015 IEEE/ACS 12 th International Conference of Computer Systems and Applications (AICCSA), p. 1-8. IEEE. find
  15. Wilkens, R., DallaVecchia, A., Boito, M. Z., Padró, M., Villavicencio, A. (2014). Size does not matter. Frequency does. A study of features for measuring lexical complexity. In: Ibero-American Conference on Artificial Intelligence (p. 129-140). Springer, Cham, (November). find
  16. Wang, Huiping., Yang, Lijiao., Xiao, Huimin. (2019). Construction of Quantitative Index System of Vocabulary Difficulty in Chinese Grade Reading. In: 2019 International Conference on Asian Language Processing (IALP), p. 480-486. IEEE. find
  17. Khoja, Shereen., Garside, Roger. (1999). Stemming arabic text. Lancaster, UK, Computing Department, Lancaster University. find
  18. Larkey, Leah, S., Margaret, E., Connell. (2001). Arabic information retrieval at UMass in TREC-10. In TREC. 2001. find
  19. Bessou, Sadik., Touahria, Mohamed. (2014). An accuracy-enhanced stemming algorithm for Arabic information retrieval. Neural Network World, 24 (2) 117. find our card
  20. Schütze, Hinrich., Christopher, D., Manning, Raghavan, Prabhakar. (2008). Introduction to information retrieval. Vol. 39. Cambridge: Cambridge University Press. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bessou-2021-efficient-measuring-readability-improve-documents/