📄 Evaluation of Machine Translation Errors in English and Iraqi Arabic

Summary

The paper tackles a gap in machine translation evaluation. Human ratings and automatic scores say how good a translation is, but not what went wrong. MITRE linguists analyzed errors from 4 English–Iraqi Arabic speech translation systems built for DARPA's TRANSTAC program, at two evaluations in 2008. They used text transcripts, so speech recognition errors were excluded.

The method borrows from HTER. Annotators post-edit a reference so that it stays as close as possible to the system output while keeping the source meaning. TER deletions, insertions, substitutions and shifts are then labeled by word class, and harmless differences are marked as null.

The proportions of error types barely changed between the two evaluations, even though the systems produced fewer errors overall. Translations into English suffered mostly from pronoun and copula problems. Translations into Iraqi Arabic were dominated by substitutions, especially subject person inflection on verbs. Polarity flips were rare.

The authors trace most errors to structural contrasts between the languages, such as dropped subjects, an absent copula and head–modifier order. These contrasts force a translation to add or drop words. Some cases, such as pronoun gender and person, require discourse context to resolve.

Key points

• Data: June 2008 and November 2008 TRANSTAC evaluations. The source utterances were:
  • 109 English→Iraqi Arabic and 97 Iraqi Arabic→English in June;
  • 93 English→Iraqi Arabic and 108 Iraqi Arabic→English in November.
• The translations analyzed came from 4 systems: 436/388 in June and 372/432 in November. Most systems were statistical; one English→Arabic engine was rule-based with a statistical fallback.
• The annotation combines a TER operation with a word class (pronoun, noun, verb, other), for example "ivp" for an inserted verb. Two extra categories were added for Arabic verb inflection:
  • Pro-Verb, for a single element combining pronoun and verb;
  • Subject Person, for wrong person inflection on the verb.
• Into English, pronouns and verbs together account for about half of all errors. In June, forms of be made up 38–52% of verb errors, depending on the system. Pronoun and copula errors together averaged over 40% of all Iraqi→English errors.
• Into Iraqi Arabic, nearly half of all errors are substitutions, and verb-related errors (including inflection) reach about 22%. Polarity reversals stay under 1% in both directions, and speech-act changes are very rare.

Interesting ideas

• I1. Build a "targeted" reference that is edited to sit as near as possible to the system output, so the remaining differences reflect real mistakes rather than legitimate paraphrase.
• I2. Mark trivial mismatches as null instead of counting every difference, using a written list: articles, number inflection, repetitions, synonyms, and an omitted and that does not change meaning.
• I3. Cross each edit operation (insert, delete, substitute, shift) with a word class to get a diagnostic error table instead of a single score.
• I4. Many errors come from material that has no counterpart in the source, such as dropped subjects, an absent present-tense copula, or resumptive pronouns that English does not need.
• I5. Choosing between "he", "she" and "it", or between first and second person on Iraqi past-tense verbs, depends on the referent and the dialogue context rather than on the sentence alone.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (back): When scoring text regenerated from the meaning graph, compare it against a minimally edited version of the output rather than the original sentence, so that paraphrase is not counted as error.
• I2 (learn): Evaluation of witness parsers and silver data could use an explicit list of harmless differences, so that the agreement metrics reflect meaningful disagreements.
• I3 (tree): Report parser errors as a matrix of UPOS × error type (attachment, label, missing or extra token), which points directly at which repair rules or training data are needed.
• I4 (back): Translation through the meta-world should explicitly decide which implicit elements (a subject, a copula) to realize or suppress in the target language, rather than mapping words one to one.
• I5 (frames): Coreference and speaker/addressee tracking in prag could supply this information, so that pronoun choice is grounded in the entity's identity rather than guessed locally.