📄 Problems of Arabic Machine Translation: Evaluation of Three Systems

Summary

The paper presents a qualitative error analysis of three free online machine translation services for Arabic and English: Google, Sakhr (its Tarjim site, run in practice through Sakhr's SET engine) and Systran. The input came from an introductory translation course at Abu Dhabi University. These were 76 English and 46 Arabic texts that grow from one-word commands to specialised paragraphs. Literary material was left out because the author considers creative writing an unfair test for MT. Outputs were judged on readability, informativeness and closeness to a human translation. Errors were then sorted into linguistic categories and traced to likely causes. In the Arabic-to-English direction, Sakhr and Google each produced fully acceptable output for about a third of texts, while Systran rarely did. Most errors came from the missing diacritics in Arabic script, gaps in the lexicon, mishandled attached clitics and proper names that were translated as ordinary words. The author argues that systems should use the lexical environment and collocations to choose equivalents, and should aim to reduce post-editing effort rather than settle for gist-level output.

Key points

• Corpus: 76 English and 46 Arabic texts from a Fall 2005–2006 translation course, ordered by rising length and complexity, with literary texts excluded.
• Fully correct Arabic→English translations: Sakhr 16 (34.7%), Google 15 (32.6%), Systran 2 (4.3%).
• Google dropped content words outright in 10 cases (21.7% of its translations), even though such words should at least have been transliterated.
• Unvocalised homographs caused many errors:
  • choosing between world and scientist;
  • choosing between active and passive verb forms;
  • picking the wrong person on past-tense verbs.
• Regional spelling variants (the Egyptian dotless final ya) and attached prefixes or pronouns also broke dictionary lookup.
• Personal and place names were rendered by their meanings. One example is the name Rasheed, which Sakhr turned into "Rosetta" and Systran into "rational". The author also judged Systran's output too literal to post-edit cheaply.

Interesting ideas

• I1. When a word went untranslated, the author removed the article, prefix or pronoun and resubmitted; the system then translated it, which showed the fault lay in clitic handling rather than the lexicon.
• I2. Some candidate morphological analyses can be ruled out on grammar alone, such as a verb-only pronoun form attached to a noun in the misparse of "Sydney".
• I3. Neighbouring proper names can serve as evidence that an ambiguous token is itself part of a name and should be kept, not translated by its meaning.
• I4. The author repeatedly suggests choosing a word's sense by checking which meaning fits the words and collocations around it.
• I5. The evaluation used graded course material, excluded poetry as out of scope and measured quality by how much post-editing a human would need.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mova's evaluation could perturb failing inputs by stripping affixes or splitting contractions to tell vocabulary gaps apart from segmentation bugs before writing repair rules.
• I2 (tokens): the morphology stage could apply hard compatibility constraints to drop impossible segmentations before tagging, so errors like this never reach the parser.
• I3 (world): entity building could use adjacent recognised names to keep surname-like common nouns as names, rather than reading them as properties or objects.
• I4 (frames): VerbNet class and role selection could score each candidate sense against its co-occurring arguments and prefer the one whose collocates fit.
• I5 (back): generated or translated text could be scored on graded inputs by estimated repair effort, giving a more practical quality measure than a simple pass/fail.