📄 DIRA: Dialectal Arabic Information Retrieval Assistant

Summary

Searching Arabic text is difficult for two reasons. The language has rich morphology, and dialects such as Egyptian Arabic (EGY) differ from Modern Standard Arabic (MSA) in vocabulary, inflection and spelling. These differences matter more now that social media carries a lot of informal, unedited dialectal writing. DIRA is a demo system that addresses this through query expansion. A user enters a lemma in English or MSA, and DIRA does three things:

• translates it into MSA and/or EGY lemmas using a trilingual dictionary,
• generates inflected surface forms with morphological generators built on analyzer databases,
• ranks those forms by corpus-based feature probabilities combined with user-set weights.

DIRA does no stemming, so its output can feed any retrieval engine. In the web demo, the user picks some of the generated forms, which are joined with Google boolean operators and sent as a query. The authors describe it as the only tool that automatically produces dialectal search terms with morphological variants. Two usage scenarios illustrate it: a teacher looking for Egyptian negation forms, and a native speaker resolving an ambiguous MSA query.

Key points

• Dictionary: lemmas are translated with a trilingual English-MSA-EGY dictionary of roughly 70,000 entries.
• Generators: forms are produced by a morphological generator (Habash, 2007).
  • For MSA it uses BAMA/SAMA databases.
  • For EGY it uses CALIMA-ARZ databases, which also yield spelling variants of the same word.
• Why MSA tools fall short: prior work cited in the paper found that over a third of EGY words cannot be analyzed by an MSA analyzer.
• Speed and ranking:
  • Expansion is made faster by a lookup cache that is first populated using sizable MSA and EGY corpora and then keeps growing online whenever new forms get generated.
  • Ranking mixes corpus lemma-feature probabilities with user weights for feature values; setting a weight to zero removes those forms entirely.
• Demo behaviour: the web demo shows up to five inflected forms per lemma, pre-selects the top two, and displays English glosses so users can tell candidate lemmas apart.

Interesting ideas

• I1. DIRA deliberately avoids stemming and outputs full inflected forms, so different downstream systems with their own normalization can all use it.
• I2. Running an analyzer's database in reverse gives spelling variants for free, because the analyzer already maps variant spellings to one conventional form.
• I3. Candidate inflections are scored by corpus frequency of feature values, adjustable by genre-specific weights, for example favouring 1st/2nd person for blogs and 3rd person for news.
• I4. A cache of lemma→forms results is seeded from corpora and grows as new forms are generated at runtime, trading memory for speed in an expensive generator.
• I5. For an ambiguous input, the system auto-picks the top reading but lists the alternatives with glosses and lets the user override, as with "harbor" versus the name Morsi.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): The en crate could expose full surface-form expansions of a lemma instead of baking in a normalization, leaving matching policy to later stages.
• I2 (tokens): Running Mova's morphology in generation mode could enumerate spelling variants (e.g., British/American, hyphenation) of a lemma, helping to match entity mentions across texts.
• I3 (back): When generating text from the meaning graph, genre-conditioned priors over feature values could steer choices of person, tense and number toward what fits the target register.
• I4 (tokens): Memoizing morphological generation and analysis results, seeded from treebank vocabulary, could speed up the tokenizer/lemmatizer without changing its output.
• I5 (world): Entity and sense resolution could keep ranked alternative readings with human-readable glosses, so a wrong pick is visible and correctable rather than silently committed.