📄 First Result on Arabic Neural Machine Translation

Summary

This short 2016 paper reports the first fully neural machine translation results for Arabic, in both directions (Ar→En and En→Ar). Earlier Arabic work used neural networks only as extra features inside phrase-based systems. The authors compare a plain attention-based encoder-decoder (the dl4mt implementation, with BPE subwords) against a standard Moses phrase-based system. Both are trained on about 1.2M mostly-news sentence pairs from LDC corpora.

They also vary Arabic preprocessing: simple tokenization, orthographic normalization, and morphology-aware ATB segmentation via MADAMIRA. English preprocessing varies too (tokenization, lowercasing or truecasing).

On in-domain news (MT05) the two systems end up close, with Moses slightly ahead. Preprocessing designed for phrase-based systems helps the neural system by a similar margin. On an out-of-domain climate-change web set (MEDAR), the neural system clearly wins in En→Ar.

This matters because it shows two things: established linguistic preprocessing still pays off for neural models, and neural MT appears more robust when the test text differs from the training domain.

Key points

• Training data: LDC2004T18, LDC2004T17 and LDC2007T08 combined give about 1.2M sentence pairs, with about 33M tokens on the Arabic side. Pairs with more than 100 tokens on either side were dropped.
• Evaluation sets: MT04 was used as dev and MT05 as test. Arabic sentences that also appeared verbatim in the training data were removed (172 from MT04, 26 from MT05). MEDAR, with four Arabic references per sentence, served as the out-of-domain En→Ar test.
• Domain distance: average log-probabilities from a 5-gram LM trained on the training corpus confirm that MEDAR is far off-domain (MT04 −59.74, MT05 −55.97, MEDAR −75.03).
• Systems:
  • NMT: a 2-layer bidirectional GRU encoder (512×2), a 512-GRU decoder, Adadelta for about seven days, dropout plus L2 (1e−4), beam width 12, and a BPE vocabulary of up to 20k.
  • Phrase-based: Moses with GIZA++, phrases up to 8 words, and KenLM.
• Best BLEU scores (Moses vs NMT):
  • En→Ar MT05: 35.98 vs 33.62.
  • En→Ar MEDAR: 19.27 vs 24.46. Preprocessing added +10.58 for Moses and +13.37 for NMT over tokenization alone.
  • Ar→En MT05: 51.19 vs 49.7.

Interesting ideas

• I1. Before scoring, the authors searched the test sets for sentences copied verbatim from the training data and removed them, so the reported scores reflect real generalization rather than memorization.
• I2. They measured domain shift directly by scoring each evaluation set with an n-gram language model trained on the training text, which turns "out-of-domain" into a number.
• I3. To undo segmentation when generating Arabic, they first look up how each segmented form appeared in the training corpus and fall back to hand-written rules only when the lookup fails.
• I4. Each preprocessing step was switched on cumulatively and reported as a gain over a minimal baseline, for two very different model types. This showed that linguistically motivated preprocessing helps both about equally.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Run the same overlap filter between silver training data and evaluation sets, and treat a high match rate as a negative control that flags inflated scores.
• I2 (learn): Score each new test genre (fables, school texts, word problems) against the training data the same way, so parser or reader drops can be tied to measured distance.
• I3 (back): When realizing text from the meaning graph, prefer surface forms already attested in the corpus for a lemma plus features, and use rule-based inflection only as a backoff.
• I4 (tokens): Run the same layered comparison of normalization and morphology options against both the perceptron parser and gparse, to learn which steps are model-independent.