📄 The ARPA MT Evaluation Methodologies: Evolution, Lessons, and Future Approaches

Summary

This paper traces how the ARPA machine translation evaluation programme changed between 1991 and 1994. The programme had to track progress across three research systems that used very different approaches and different language pairs, and it also had to compare them with commercial production systems. A further difficulty was that human judgments of translation quality are inherently subjective. The authors explain why several early methods were dropped: back-translated comprehension tests, expert quality panels using a government translator-grading scale, and human-assisted (HAMT) output. Each either confounded results with factors outside the core engine or was too hard to run at scale. These methods gave way to three measures that monolingual English speakers can apply: adequacy, fluency, and informativeness. Analyses of variance on the January 1994 data showed that evaluator, text, domain, and booklet order all had significant effects. The August 1994 design responded by controlling these factors and collecting many more judgments. The work matters because it laid down practical principles for repeatable, portable, low-bias MT evaluation.

Key points

• The three research systems were:
  • CANDIDE (IBM), statistical, French→English.
  • PANGLOSS (CMU, NMSU, USC ISI), knowledge-based and linguistic, Spanish→English, with Japanese→English added in August 1994.
  • LINGSTAT (Dragon Systems), hybrid, Japanese→English, with Spanish→English added in August 1994.
• The standard suite settled on three components:
  • Adequacy: fragment-level checks against a professional reference translation.
  • Fluency: sentence-level judgments made without a reference.
  • Informativeness: multiple-choice comprehension questions.
• ANOVAs on January 1994 results found significant effects of four methodological factors: evaluator, text, domain, and text order in the booklets.
• The August 1994 evaluation used only fully automatic output (FAMT) and 100 general-news source texts per language. It obtained two sets of expert ("level five") translations: half served as adequacy references, and the rest were scored like system output.
• The number of judgments per system grew roughly fivefold:
  • Informativeness: 120 → 600.
  • Fluency: about 347 → 1735.
  • Adequacy: 605 → 3025.

Interesting ideas

• I1. Constituent-sized adequacy units. Adequacy is scored on syntactic fragments shorter than a sentence. This avoids favouring systems that do well only on long clauses or only on short spans, a useful precaution whenever compared systems differ architecturally.
• I2. Fluency judged without a reference. Separating fluency from content accuracy makes raters judge well-formedness on its own terms. The same split is useful for evaluating any text generation output.
• I3. Counterbalanced booklets. Booklets vary the order of components, show raters different texts in each component, and rotate which system appears first. This limits halo, context, and fatigue effects and is directly reusable in crowdsourced annotation.
• I4. Re-scored anchor texts. Old outputs with known scores are re-evaluated as practice items. They serve to train raters, calibrate new raters against earlier cohorts, and measure progress over time, a cheap way to keep longitudinal benchmarks comparable.
• I5. Scoring human references like systems. Expert translations go through the same measures as machine output. This gives a human ceiling and exposes how noisy the metric is, which helps interpret system scores realistically.