📄 Parallel resources for Tunisian Arabic Dialect Translation

Summary

The paper addresses the lack of parallel data for translating Tunisian Arabic dialect (TD), as written informally on social media, into Modern Standard Arabic (MSA). The authors assembled about 9.7K TD-MSA sentence pairs from three existing sources (PADIC, MADAR, and an aligned TD/MSA version of the Tunisian constitution, TD-CONST) and from 900 Facebook comments (TD-COM) translated by a native speaker. From word alignments they derived a manually corrected bilingual dictionary. To augment the data, they split MSA sentences at stop words and punctuation, then used the dictionary to translate each MSA fragment back into TD, which produced new fragment-level pairs. A phrase-based SMT system was trained on several corpus configurations and tested on held-out social-media comments. Training on the original pairs together with the fragment pairs gave the best BLEU (15.03%). Training on the fragments alone gave the worst result. The work matters as an early, inexpensive recipe for a dialect that previously had almost no translation resources.

Key points

• Corpus sizes: PADIC 6.4k lines, MADAR 1.8k, TD-CONST 600 (after manual segmentation of about 500 law texts), TD-COM 900. The total is 9.7k pairs, with about 68k TD words and 74k MSA words.
• The TD-MSA dictionary was built from GIZA++ HMM word alignments and then manually cleaned. It has 44k entries.
• On average only about 30% of word types are shared between TD and MSA. The authors take this as evidence that the two should be treated as separate languages.
• The SMT system is Moses-style, with IRSTLM language models. The test set is 500 TD-COM comments, and the remaining 400 are used for training.
• BLEU scores by training set:
  • PADIC+MADAR: 13.07
  • adding TD-CONST: 12.00
  • PADIC+MADAR+TD-COM: 13.67
  • all four sources: 13.27
  • segmented only (32k lines): 11.58
  • original plus segmented (41k lines): 15.03

Interesting ideas

• I1. Synthetic fragment pairs (made by cutting the target side into pieces and back-translating each piece with a dictionary) helped only when added alongside the original sentences, and hurt when used alone. Augmentation should extend real data, not replace it.
• I2. Adding more data from a distant register (the legal constitution text) lowered quality on the social-media test set. Domain proximity mattered more than raw volume.
• I3. Measuring the overlap of word types between two language varieties gives a simple, explainable estimate of how far apart they are.
• I4. A bilingual lexicon was bootstrapped from automatic word alignment and then cleaned by hand. This yields a large resource cheaply while keeping it trustworthy.
• I5. Splitting long sentences at function words and punctuation produces shorter, more learnable units for a low-resource model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mix silver or synthetic training items with gold ones instead of substituting them. Run a "synthetic only" negative control to catch damage from degraded data.
• I2 (learn): When growing silver corpora for school tests or fables, check each added source against an in-domain evaluation set before keeping it.
• I3 (tree): When converting treebanks between UD dialects or registers, a vocabulary-overlap score could indicate how much lexical normalization a conversion step needs.
• I4 (world): Compiled lexical knowledge, such as verb-class or synonym mappings, could be drafted automatically from aligned data and then pruned by review or an LLM teacher before entering the knowledge layer.
• I5 (back): The text → graph → text compressor or translator could be trained on clause-level fragments as well as full sentences, which gives more aligned examples per document.