📄 Challenges in Processing Colloquial Arabic

Summary

The paper asks whether the large text resources for Modern Standard Arabic (MSA) can improve language models for spoken Egyptian Colloquial Arabic, which has very little transcribed data. The work targets speech recognition and measures quality with test-set perplexity. The authors train trigram models on the CallHome ECA corpus and on MSA newswire from Arabic Gigaword, then test three ways to make MSA text resemble the dialect:

• stemming both training and test data;
• a hand-built Egyptian-to-MSA lexicon decoded through stem lattices;
• automatically discovered reorderings of constituents in Penn Arabic Treebank trees.

Each technique lowers the perplexity of the MSA model on Egyptian text. Stemming cuts it roughly in half, and the lexical mapping brings the total reduction to about 82%. The syntactic reorderings give only small gains. None of these models beats a model trained on the small dialect corpus alone. Adding more MSA data even makes results on Egyptian worse. The authors conclude that, for language modeling, MSA behaves like a separate language from the dialect. This is a useful negative result for anyone planning cross-variety transfer.

Key points

• Data:
  • CallHome Egyptian Colloquial Arabic: 130K training tokens and a 32K-token test set, used in Arabic script.
  • LDC Egyptian lexicon: 51,202 entries.
  • Arabic Gigaword: mainly the AFA portion, about 97M tokens.
  • Penn Arabic Treebank: tagged with Buckwalter's analyzer.
• Models: SRILM trigram models with Good-Turing discounting and Katz backoff.
• Baselines:
  • An Egyptian-trained model reaches perplexity 184.8 on Egyptian test data.
  • A same-size MSA model scores 12874.2 on the same data, about 65 times worse.
  • On its own newswire domain, MSA perplexity falls from 955.4 to 157.4 as training grows from 130K to 27M tokens. On Egyptian data, MSA perplexity rises as training grows.
• Vocabulary: at equal size (130K tokens), MSA has about 30,000 word types against 13,500 for ECA, and 95,000 bigram types against 60,000.
• Transformation results on Egyptian test data:
  • Word-based MSA model: 12874.2.
  • Stemming: 6260.7.
  • Stemming plus lexical transductions: 2262.3. The lexicon covered about 82% of ECA tokens.
  • Syntactic transformations: about fifty useful ones, each giving only a 5–8% reduction.
• English comparison: adding 227M words of NAB newswire to CallHome English raises test trigram coverage from 34% to 72.5%. For Egyptian, adding MSA brought almost no coverage gain.

Interesting ideas

• I1. If a source corpus really matches the target, perplexity should fall as you add more of it; a rising curve signals mismatch, and the authors check this against a clearly unrelated control (English NYT text evaluated on Egyptian).
• I2. Instead of committing to one standard equivalent per dialect word, each word becomes a lattice of candidate standard stems, and a language model picks the most likely path through the sentence.
• I3. Candidate syntactic rewrites are generated by permuting the children of frequent tree productions, and only those that measurably help the target data are kept.
• I4. N-gram coverage of the test set is used as a cheap check on whether extra out-of-domain text could help at all, run before any expensive training.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when adding new silver data, plot target-set accuracy against data size and include a deliberately unrelated corpus as a negative control, so "more data hurts" is caught early.
• I2 (tokens): normalize slang, misspellings, or regional English forms into a lattice of candidate standard lemmas, and let a context model choose, keeping the alternatives visible for explanation.
• I3 (tree): when converting between treebank or UD dialect conventions, generate candidate restructuring rules automatically and keep the ones that raise agreement with a small gold target sample.
• I4 (learn): before training on a new corpus, compute n-gram or subtree coverage of the evaluation genre (fables, school tests) to predict whether it is worth adding.