📄 Comparative Evaluation of Arabic Language Morphological Analysers and Stemmers

Summary

Arabic stemmers and morphological analysers have each been tested on their author's own data, with their own scoring rules, so their published accuracies cannot be compared. Sawalha and Atwell address this by building two hand-annotated gold standards of about 1000 words each. One is chapter 29 of the Qur'an (Al-Ankaboot); the other is newspaper text from the Corpus of Contemporary Arabic. Arabic language experts verified the roots in both. They define four scoring schemes: tokens or word types, each with and without stop words. Three freely obtainable systems are scored under all four: the Khoja stemmer, Tim Buckwalter's analyser of Arabic word structure, and an algorithm designed to pull out tri-literal roots. Their outputs are also combined by majority voting. All four schemes give the same ranking: Khoja first, then tri-literal, then Buckwalter. Voting matches or slightly beats the best single system, reaching roughly 62% on the Qur'an text and 70% on news. The authors conclude that Arabic root extraction is too unreliable to feed POS tagging and parsing, because errors would propagate downstream.

Key points

• Gold standards: 978 Qur'an tokens (616 word types; 554 tokens without stop words) and 1005 newspaper tokens (710 types; 766 tokens without stop words), drawn from the Al-Rai paper in the Corpus of Contemporary Arabic.
• Token accuracy including stop words: Khoja 68.2% (Qur'an) and 77.01% (news); tri-literal 59.71% and 76.72%; Buckwalter 57.16% and 40.70%.
• Ties in voting were broken two ways. Experiment 1 picks at random using Python FreqDist. Experiment 2 takes the answer of the most accurate system, and this version scored better (e.g. 64.4% vs 60.7% on Qur'an word types).
• All systems performed better on newspaper text than on Qur'anic text, which is consistent with their original design for news.
• Only three systems were evaluated, because other stemmers were not freely available; the authors call for open-source sharing of Arabic resources.

Interesting ideas

• I1. Score every system under several views of the same data (token vs type, with vs without function words), so a ranking that holds across all views is trustworthy and one that flips is exposed.
• I2. Combine several imperfect analysers by per-word majority vote, and break ties by deferring to the system that measured as most accurate, rather than choosing at random.
• I3. Build small (~1000-word) expert-checked gold sets from two contrasting genres, so that genre sensitivity shows up directly in the comparison.
• I4. Agreement across all systems tends to happen on easy cases, so value lies in the disagreements.
• I5. Upstream morphology errors compound into tagging and parsing, so stem/lemma accuracy should be measured before trusting later stages.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Report lemma and tagger accuracy on both tokens and distinct types, and with and without closed-class words, so frequent easy function words cannot hide weakness on content words.
• I2 (learn): When the witness parsers disagree on a silver-data label, a tie should go to the historically strongest witness; this paper shows that rule beats random tie-breaking.
• I3 (tokens): A compact lemma/morphology gold set split between modern prose and older fable or tale language would reveal where the en crate's lemmatiser degrades by genre.
• I4 (learn): Words where all witnesses agree can be auto-accepted, while disagreement items are routed to the LLM teacher tier or to repair-rule induction.
• I5 (tree): Track how lemma and morphology errors propagate into LAS, so that parser failures caused by bad morphology are explained rather than blamed on the parser.