📄 Inductive coding of the Arabic lexicon

Summary

SYSTRAN began building Arabic, Farsi and Urdu to English MT systems in July 2002 and had few trained Arabic lexicographers. This paper explains how it made dictionary coding easier for native speakers without linguistic training. The method is SYSTRAN's IntuitiveCoding® technology (ICT). Lexicographers enter surface stems, not roots and patterns. Entries stay in the native script, with no transliteration, and live in an SQL database that keeps versions.

A statistical stem-guesser proposes the stems for each entry, and the coders accept or correct them. Two separate declarative modules then handle morphology:

• Internal morphology expands the stems into a full inflected dictionary, which is compiled into finite-state automata.
• External morphology splits clitic-laden tokens into conjunctions, prepositions, the core word and pronoun suffixes before dictionary lookup.

The resulting Arabic-English system has more than 30,000 single-word entries and over 90% terminology coverage on news and web text, with gisting-level quality. The work matters because it shows how cheap, guessed-then-verified lexicon building can quickly bootstrap a morphologically rich language.

Key points

• The lexicon is stem-based rather than root-based. Verbs and their derived nominals are separate entries joined by a link, and a validator checks consistency, for example that the root survives across a verb's stems.
• Each entry has six fields plus notes: lemma, part of speech, type, morphological, syntactic and semantic information. Nouns have five types and adjectives two (base and comparative).
• The guesser uses only the rules that cover the most forms in a category, so it overgeneralizes. Even so, it made lexicographers about 60% more productive.
• Internal morphology works at the paradigmatic level and handles hollow and weak verbs, duals, regular plurals, deletion and epenthesis. External morphology works at the syntagmatic level, runs first and feeds lookup, and uses pattern rules such as WAFA and KABILI.
• The authors report 30,000+ entries, over 90% coverage of newspaper and Internet terminology, and gisting-level output. A compound morphology module is planned.

Interesting ideas

• I1. A frequency-based guesser pre-fills each entry and a human only fixes the misses; accepting a known overgeneralizing guesser still saves much effort, because review is cheaper than writing from scratch.
• I2. Irregular stems are stored directly in the dictionary, so the inflection rules stay simple and only cover regular processes.
• I3. Splitting a token into separate words (syntagmatic) is kept apart from inflecting a single word (paradigmatic), and splitting runs before lookup while keeping multiple parses for later disambiguation.
• I4. Derived entries are distinct but linked, and an automatic validator checks invariants across the links instead of deriving everything from an abstract base.
• I5. The dictionary database saves versions so translation quality can be compared across lexicon releases.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Unknown English words could get morphology and lemma proposals from majority rules, with an LLM teacher tier or human only confirming or correcting them, and the correction rate tracked.
• I2 (tokens): The en crate could list irregular principal parts (went, gone, mice) as lexicon fields and keep the generator rules purely regular, which makes every generated form easy to trace.
• I3 (tokens): English clitics like 's, n't, 'll and 're could be split by explicit patterns before lexicon lookup, with ambiguous cases such as 's (is/has/possessive) kept as alternatives for the tagger.
• I4 (world): Verb–nominalization pairs from WordNet derivational links could be stored as separate linked entries, with a check that the nominal's frame keeps the verb's roles.
• I5 (learn): Lexicon and rule snapshots could be versioned and the evaluation suite rerun per snapshot, so each regression can be traced to a specific entry change.