📄 The Role of Diacritics in Increasing the Difficulty of Arabic Lexical Recognition Tests

Summary

Arabic lexical recognition tests (LRTs) estimate learner vocabulary size by asking which items are real words. Earlier Arabic tests left out diacritics, and they turned out too easy: a learner who recognises the bare root passes, which produces ceiling effects. The authors ask whether diacritized forms, and especially rare ones, can make such tests harder in a controlled way.

To find the frequency of each diacritized form, they built a pipeline over a freely available, topically diverse web corpus (Freihat et al., 2018). Farasa adds diacritics, segments words and lemmatizes them. Clitics are then removed, and DKPro Core tags roots with their diacritics so that each form can be counted.

They then ran a web-based user study with 263 participants. Each person was randomly given one of three versions of the same test: no diacritics, the most frequent diacritization, or the least frequent one. Mean balanced scores fell from 91.8 to 86.8 to 82.3, and all differences were significant. Only the rare-diacritics version showed no ceiling effect. This makes it the most promising version for measuring advanced learners.

Key points

• The test has 40 words and 20 nonwords. The nonwords were made by substituting or transposing letters in real words. The base items come from "test A" of Hamed and Zesch (2017b).
• The pipeline is Farasa (diacritization, segmentation, lemmatization), then clitic removal, then DKPro Core (UIMA-based) to label roots with diacritics and count them. Farasa was chosen over Madamira based on the authors' earlier benchmarks.
• The source corpus mixes several kinds of text:
  • Aljazeera (30%)
  • Arabic Wikipedia (20%)
  • novels (15%)
  • Alquds newspaper and Altibbi (10% each)
  • IslamWeb, social networks and other sources (5% each)
• Participants split across the three versions as 96 (no diacritics), 78 (frequent diacritics) and 89 (infrequent diacritics). Word precision stayed similar across versions (.95/.91/.92), but word recall dropped to .80 under infrequent diacritics. A teacher review and an 11-student pilot came before the main study. The PHP/MySQL web interface is released at github.com/ohamed/ar-lrts.
• Item analysis showed an uneven effect. Some items became much harder with rare diacritics; for example, scores for I* fell to .31, Ox to .25 and mdp to .41. Others barely changed. Corpus frequency did not clearly explain which items were affected.

Interesting ideas

• I1. Keep the lemma fixed and swap in a rarer but valid surface realization; this raises difficulty in a controlled way without inventing new content, so you can tune difficulty along one axis.
• I2. Score as the average of recall on real items and recall on distractors, so an always-yes strategy and an always-no strategy are punished equally whatever the class sizes.
• I3. When gold annotation is scarce, get frequency statistics by running the best-benchmarked automatic annotator over a large raw corpus, after first checking that its output yields trustworthy counts.
• I4. Do not stop at aggregate scores; a per-item view (here a heatmap across conditions) can show that an overall effect comes from a few items, and that the obvious explanation (frequency) does not hold.
• I5. Build plausible distractors by transposing or substituting letters in real words, which gives near-miss nonwords that probe genuine recognition.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): School-course test generators could make harder variants of a question by choosing rare word forms or senses of the same lemma. This would show whether a reader truly knows the word or only its common form.
• I2 (learn): Evaluation with negative controls could use this balanced recall. A component that accepts everything, or rejects everything, then cannot look good just because one class is larger.
• I3 (learn): Lemma and form frequencies could come from silver-annotated English text. Before the counts are trusted, they would be checked against a small gold sample.
• I4 (learn): Evaluation reports could break results down per item or per construction. This would show which inputs cause a regression instead of hiding them inside one averaged number.
• I5 (tokens): Such pseudo-words could stress-test the tagger and morphology on unknown words. The check is whether Mova flags them as unknown rather than confidently forcing a lemma onto them.