📄 Evaluation of machine translation and its evaluation

Summary

The paper asks how to judge automatic MT metrics when the human judgments used as ground truth are themselves inconsistent. The authors recast MT scoring as standard precision, recall and F-measure. They define the overlap between a candidate and a reference as a maximum matching of identical words in a bitext grid. An exponent e lets the score reward contiguous, correctly ordered runs more than isolated hits. Multiple references are handled by concatenating them and capping the match at the mean reference length.

The method was tested on 2002 DARPA MTEval data: Arabic→English and Chinese→English translations scored by humans for Adequacy and Fluency. Correlation with those scores was measured on pseudo-documents of different sizes.

Human judges agreed with each other surprisingly poorly, and every automatic metric fell further below human-level reliability. Even so, the unigram F-measure (e = 1) correlated significantly better with humans than BLEU and NIST. It is also easier to interpret, because it can be read as the share of the grid covered by aligned blocks. The software, GTM, is publicly released.

Key points

• Data:
  • Arabic: 728 source sentences with 10 English versions (6 system outputs, 4 human references).
  • Chinese: 878 source sentences with 14 English versions (10 system outputs, 4 human references).
  • Each candidate sentence was rated by humans on a 1–5 scale for Adequacy and Fluency.
• Low human agreement: Two judges who both rated the same 227 sentences reached a Spearman correlation of only 0.019 on Adequacy.
• Approximate matching: With e > 1, finding the maximum matching is conjectured to be NP-hard. A greedy method that repeatedly adds the largest non-conflicting block was used instead. It found an optimal matching in 99% of simulated cases, and otherwise reached at least 80% of the optimum.
• Metric comparison: On the Chinese corpus and the lowercased, stemmed Arabic corpus, the F-measure with e = 1 outperformed BLEU and NIST. Significance was checked with the Wilcoxon signed ranks test.
• Counterintuitive effects: In some settings, extra references and longer documents lowered correlation. Notably, BLEU did worse with three references than with one on longer documents. The authors attribute this to "distracting" n-grams. GTM is released under a BSD-style license.

Interesting ideas

• I1. Defining overlap as a maximum matching: Treating the overlap between two texts as a maximum matching in a word grid prevents double-counting repeated words. The same trick could give clean precision/recall for any text-vs-reference task, such as summarization or text generation.
• I2. A tunable word-order bonus: Weighting each matched run by its length raised to a power e, then taking the e-th root, gives one knob for how much contiguous order matters. e = 1 ignores order, while larger values favour fluent output. This beats hard-coding n-gram counts.
• I3. Fair use of multiple references: References are concatenated with barriers so runs cannot cross from one reference into another. The match is then trimmed to the mean reference length, removing hits from the shortest runs first. This keeps a candidate from scoring well by picking words scattered across many references.
• I4. Testing metrics at many text sizes: Building pseudo-documents from random sentences and checking metric–human correlation at each size shows whether a metric is trustworthy on small units. That matters for error analysis, where developers inspect individual sentences rather than corpus averages.
• I5. Use human agreement as the ceiling: Plotting inter-judge correlation next to the metric curves shows how much agreement is achievable at all. It also warns against metrics whose parameters are tuned to one corpus, which can rank systems misleadingly on new data.