📄 Modifying a Natural Language Processing System for European Languages to Treat Arabic in Information Processing and Information Retrieval Applications

Summary

The authors describe adapting LIMA, a modular analyzer at CEA LIST built for English, French, German and Spanish, to Arabic, a Semitic language. Two Arabic properties broke the original design. First, printed text is often only partly voweled or not voweled at all, while dictionary headwords carry vowels. Second, articles, prepositions, conjunctions and pronouns attach to words as clitics.

For clitics, the team did not store every agglutinated form. Instead they kept a large clitic-free full-form lexicon, added two small clitic lists, and inserted a dedicated clitic-stemming step into morphological analysis. For missing vowels, they reused the mechanism that already mapped unaccented European words to accented entries.

The adapted pipeline was plugged into an online cross-language retrieval system covering Arabic, English, French and Spanish. The integration took roughly one person-year once the lexicon and training corpus existed. The paper reports no quantitative evaluation; TREC 2001/2002 experiments are only planned.

The case shows that a modular platform can absorb a new language family through targeted additions rather than a redesign.

Key points

• The Arabic full-form lexicon has 5.4 million entries. The proclitic list has 77 combinations and the enclitic list has 65, each stored in voweled and unvoweled variants. Generating every clitic-bearing form would yield an estimated 60 billion forms (Attia, 1999).
• The clitic stemmer works in four stages:
  • it normalizes some vowel marks and letter variants;
  • it proposes every proclitic/enclitic split and looks up the remaining radical;
  • if the lookup fails, it applies about a dozen rewrite rules in the spirit of Darwish (2002);
  • it keeps only splits whose three parts have mutually compatible tags.
• The existing German compound splitter was rejected for this task. It allows any dictionary word as a component, which would add noise, whereas Arabic clitics form a small closed set.
• Because Arabic entries are voweled and input text may not be, the 482 idiom rules are written over lemmas rather than surface forms. Proper names remain a weak point: there are no capitalization cues, and names are sometimes wrongly stemmed or tagged as verbs.
• The demo at alma.oieau.fr holds 50 documents per language. Results are returned in ranked classes defined by which query subgraphs they match; the example query's top three classes contain 14, 18 and 9 documents.

Interesting ideas

• I1. Clitic splitting is driven by short closed lists of attachable pieces, and every candidate split must pass a lexicon lookup and a tag-compatibility check; this avoids open-ended decomposition that floods the word graph.
• I2. A component built for a new language was turned back on an old one: the Arabic clitic stemmer, given a Spanish enclitic list, finally handled forms like "dame".
• I3. Analysis keeps competing readings as alternative paths in a word graph and leaves pruning to the later tagging step, so early stages never commit too soon.
• I4. Retrieval indexes nested sub-chunks of a compound (whole phrase plus head–modifier pairs) and ranks documents in classes by how completely the query's structure is matched, not by term counts.
• I5. Translated variants of a compound are kept only when they actually occur in the target-language index, which filters out implausible translations.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): English contractions and clitics ('s, n't, 'll, gonna) could be split from a closed list, with a split accepted only when host and clitic tags agree.
• I2 (tokens): a mechanism added for one hard phenomenon should be tested against other phenomena of the same shape, such as reusing a contraction splitter for possessives and fused informal forms.
• I3 (tree): the tokenizer could pass an ambiguity lattice to the parser and record which path the tree chose, making the disambiguation decision explainable.
• I4 (answer): passages in a fable could be ranked by how many of the question's frame pieces (agent, action, object) they match, giving graded, explainable relevance.
• I5 (back): candidate renderings generated from the meaning graph could be checked against attested phrases in a corpus before being emitted.