📄 MT and Arabic Language Issues

Summary

This AMTA 2012 tutorial by Nizar Habash surveys the main obstacles Arabic poses for machine translation and the practical remedies used around that time. It is aimed at MT researchers and translators and assumes no knowledge of Arabic. It is a review rather than a single experiment, organised by linguistic layer:

• Script and spelling: right-to-left writing, letter shaping, optional diacritics, hamza variants, encodings and romanization.
• Morphology: root-and-pattern derivation, clitics, ambiguity and tokenization for MT.
• Syntax.
• Dialects: spoken, unstandardised and poorly resourced varieties.

The tutorial first recaps phrase-based SMT concepts such as GIZA++ alignment, phrase tables, decoders like Moses, BLEU, pivoting and OOV words. It then gives a practical checklist: clean the encoding, strip tatweel, and normalize spelling variants. It also introduces Columbia's CADIM dialect toolset and the CODA orthography convention. The main reported result comes from Elissa, a rule-based system that rewrites dialectal Arabic into MSA. Used as a pre-step before an MSA-English Moses system, it gives a modest BLEU gain. The deck matters as a compact map of where Arabic MT breaks and which preprocessing choices fix it.

Key points

• Diacritics are mostly absent from written Arabic. In news text only about 1.5% of words carry any diacritic, and a word averages about 6.8 possible diacritizations. A single glottal-stop phoneme can be written with six different characters.
• The tutorial distinguishes two kinds of normalization:
  • Reductive normalization collapses Alif, Ya, Ta Marbuta and hamza variants, and suits internal models or non-Arabic targets.
  • Enriching normalization picks the correct form in context (e.g. with MADA), and is needed when Arabic is the output.
  • REMOOV is suggested for reducing OOVs.
• Buckwalter transliteration is a reversible one-to-one romanization of Arabic letters, used in the Penn Arabic Tree Bank. Unicode is the recommended encoding, and different schemes should never be mixed.
• The CADIM tools include AIDA, CALIMA, CODAFY, MADA-ARZ, ELISSA, DIRA and the Tharwa Egyptian-MSA-English dictionary of roughly 55K entries.
• Elissa uses the ADAM analyzer, hand-written transfer rules and an MSA language model. Placed before a Moses MSA-English system trained on 64M words, it raised BLEU as follows:
  • dev set: from 37.20 to 37.86;
  • blind test: from 38.18 to 38.80. The best setup rewrote only MT OOVs and dialect-only words.

Interesting ideas

• I1. How aggressively to normalize depends on direction: merge spelling variants freely when the text is only analysed, but restore the right form in context when the text must be produced.
• I2. Elissa rewrites only the words the downstream system cannot handle (OOVs and dialect-only forms) and leaves everything else untouched, which gave its best score.
• I3. CODA was designed against explicit goals: one convention across all varieties, consistency, easy to learn, and readable without training.
• I4. Measuring ambiguity directly, as the average number of analyses per word, shows how hard disambiguation really is, beyond the intuition that "no vowels means very ambiguous."
• I5. The tutorial notes that BLEU penalises synonyms and inflectional variants, and that ad hoc spellings of foreign names fragment the data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): keep a collapsed key (curly vs straight quotes, variant spellings) for lexicon lookup while preserving the original surface form for the tree and for generation.
• I2 (tree): before parsing, map only non-standard tokens (informal or archaic spellings in tales) to standard forms, and log each mapping so the parse stays explainable.
• I3 (learn): when converting treebanks between UD dialects, write such goals down first and test silver data against them instead of making ad hoc per-corpus choices.
• I4 (tokens): report mean candidate analyses per token before and after tagging, so morphology changes can be judged by how much ambiguity they remove.
• I5 (back): score text → graph → text output by comparing meaning graphs or lemmas rather than surface n-grams, and merge name-spelling variants into one entity in the world before generating.