📄 Constructing and Using Broad-coverage Lexical Resource for Enhancing Morphological Analysis of Arabic

Summary

The paper addresses a gap in Arabic NLP. Classical Arabic lexicons hold a great deal of morphological knowledge, but they are not stored as computational databases. They also follow idiosyncratic ordering schemes, mostly root-based entries with long, unstructured definition articles. The authors gathered 23 freely available machine-readable lexicons spanning roughly 1200 years and merged them into a single broad-coverage resource intended to support lemmatizers, morphological analyzers and POS taggers. Building it took three steps. First, each lexicon was manually normalized to its most frequent entry format. Next, dedicated programs pulled out roots, derived words and meanings, pairing every word found in a definition with the root that heads it. Finally, a combination algorithm merged the results. Coverage was evaluated against three test sets: the Qur'an, a portion of the Arabic Web Corpus, and the Corpus of Contemporary Arabic. Exact surface matching reached only about two thirds, because Arabic attaches clitics and affixes. When coverage was measured through a lemmatizer that strips clitics before lookup, it rose above 80%. This shows that a lexicon's usefulness depends on the morphology layer that sits in front of it.

Key points

• Sources: 23 free lexicons, most of them MS-Word files from the al-miškāt Islamic library. Together they total 247 files (178.32 MB) and 14,369,570 words, with 2,184,315 vowelized and 569,412 non-vowelized word types.
• Traditional lexicons follow four ordering traditions:
  • al-ẖalῑl: phonetic order by point of articulation.
  • abῑ ‘ubayd: grouped by topic.
  • al-ğawharῑ: ordered by last letter, then first letter.
  • al-barmakῑ: an alphabetical re-sort of aṣ-ṣiḥāḥ.
• The shared structure across all sources is root followed by definition. Extraction produces word–root pairs from each definition block.
• Exact word matching against the test corpora gave about 65–68% coverage. Lemmatizer-based coverage gave about 85% with function words included and about 82% without them.
• The lemmatizer also relies on three auxiliary lists: function words (Diwan, 2004), named entities (Benajiba et al., 2008) and broken plurals. It is intended for the 100-million-word Arabic Web Corpus.

Interesting ideas

• I1. Coverage is judged in two ways: by raw string lookup, and by whether a real consumer (a clitic-stripping lemmatizer) retrieves a correct analysis, which gives a far more honest usefulness number.
• I2. Root-organized dictionary articles are treated as weak supervision, with every word inside a definition block linked to the heading root to build derivational families cheaply.
• I3. Coverage is reported both with and without closed-class function words, since those easy, fixed-analysis items can mask weakness on content words.
• I4. Sources with different layouts are first normalized to each source's dominant format, then parsed by source-specific extractors, and only afterwards merged by one combination step.
• I5. The pooled dictionary text is itself reused as a corpus from a different domain than the usual news-based corpora.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Measure the lexicon in crate en by how often the lemmatizer produces a valid lemma and analysis on running text, not by surface-form hits, so that inflection gaps and lexicon gaps are told apart.
• I2 (world): Mine WordNet glosses or similar headword-grouped sources to assemble derivational clusters (write/writer/writing), which could link entities and events across part-of-speech changes in the world model.
• I3 (learn): Report tagger and lemmatizer metrics split into content-word and function-word slices, as a built-in control against inflated aggregate scores.
• I4 (world): Apply the same normalize–extract–merge staging when combining VerbNet, FrameNet, WordNet and Wikidata, keeping each source's extractor separate and explainable before unification.
• I5 (tokens): Dictionary definitions and usage examples could serve as an out-of-domain stress test for the tagger and lemmatizer, exposing failures that news-style treebanks hide.