📄 OSMAN — A Novel Arabic Readability Metric

Summary

Standard readability formulas such as Flesch and Fog depend on syllable counts, which are hard to obtain for Arabic. Arabic text is usually written without diacritics, and those diacritics are what determine syllables and pronunciation. Earlier Arabic metrics either ignored syllables entirely or estimated them crudely. The authors present OSMAN, an open-source Java tool and metric that adapts Flesch/Fog-style formulas to Arabic. It counts short, long and stress syllables from diacritics. When text lacks diacritics, the tool first restores them with Mishkal. It also adds a "Faseeh" factor based on letters and affixes that informal or dialect-influenced writing tends to drop or misspell.

The metric was evaluated on about 73,000 parallel English–Arabic sentences from UN General Assembly resolutions. The working assumption was that a professional translation should keep readability rankings intact. On this data, OSMAN applied to diacriticised Arabic correlated well (Spearman) with English Flesch. The same metric applied to undiacritised Arabic did not. The language-independent measures ARI and LIX could not tell the two Arabic versions apart.

Key points

• Data: a UN parallel corpus (Rafalovitch and Dale, 2009).
  • Roughly 3 million words per language from over 2,000 documents.
  • Converted from TMX into files tagged with docID, sentID and lang, and released publicly.
• Diacritics: restored with Mishkal, whose reported accuracy is above 85%.
  • Syllables are counted from four vowel-related diacritics plus Tanween.
  • Word length is measured with diacritics excluded.
• Corpus statistics: across two million Arabic words, the average word is about five letters long and has about four syllables. Many syllables therefore do not make an Arabic word complex.
• Undiacritised text misleads: OSMAN on undiacritised Arabic gave an inflated "easy" mean of 88.49.
• Human check: five native speakers per language rated ten documents on a 5-point Likert scale.
  • Agreement with OSMAN was high for diacriticised Arabic.
  • Agreement was low for undiacritised Arabic.

Interesting ideas

• I1. Validating a metric through translation: when a translation is of high quality, the ranking of a text-level score should survive the change of language. This lets a metric be checked without any gold labels.
• I2. Rebuilding the hidden layer before measuring: the authors restore the information that the script omits (diacritics) instead of computing features from the impoverished surface form.
• I3. Building features from documented learner errors: the Faseeh factor comes from an empirical study of the spelling mistakes students actually make. Error-prone spots are a principled source of difficulty cues.
• I4. Checking whether a metric reacts to a known degradation: ARI and LIX scored text with and without diacritics the same way, which exposed their blindness to the missing information.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): rank-correlate per-sentence complexity or structure scores across parallel texts as a label-free sanity check on parser-derived measures.
• I2 (tokens): base complexity features on restored morphology and lemma or syllable analysis rather than on raw character length.
• I3 (answer): when grading school-course texts or tests by difficulty, weight the constructions that learners demonstrably get wrong.
• I4 (learn): add perturbation-based negative controls (e.g., stripped morphology, shuffled words) and require each metric or model to register the damage.