📄 Precision and Recall of Machine Translation

Summary

The paper argues that machine translation output can be scored automatically with the familiar information-retrieval measures precision, recall and their combination, the F-measure. This is an alternative to BLEU, whose behaviour the authors consider hard to interpret. Candidate and reference words are laid out on a bitext grid, and overlap is defined as the size of a maximum matching: each word can be paired at most once. To reward correct word order, the matching size is generalised so that longer contiguous runs of matched words count more, controlled by an exponent. Several reference translations are handled within the same grid framework. The authors compared their measures with BLEU by correlating both against human judgments of translation quality, using Pearson and Spearman coefficients on pseudo-documents of different sizes. The F-measure correlated with human judgments at least as well as BLEU, and often better. It also depended less on how many references were available. This matters because it offers an interpretable and well-understood metric for MT evaluation.

Key points

• Overlap is the maximum matching size in a bitext grid. Unlike plain n-gram counting, this prevents any word from being credited twice.
• A run-length generalisation weights longer contiguous matched sequences more heavily via an exponent. Setting the exponent to 1 gives a plain unigram overlap.
• Evaluation correlated metric scores with human judgments using both Pearson and Spearman coefficients. Pseudo-documents contained 2, 3, 5, 10 or 15 segments, and 1, 2 or 3 reference translations were used.
• In the reported plot, F-measure generally matches or exceeds BLEU's correlation. Its scores with a single reference are competitive with BLEU's scores using more references.
• The authors released an implementation of the measures as publicly available software.

Interesting ideas

• I1. Reuse standard measures instead of inventing a new score. Precision, recall and F-measure have well-known interpretations, so results are easier to explain and debug. This could carry over to evaluating other generation tasks, such as Ukrainian summarisation or transliteration.
• I2. Matching instead of counting. Framing overlap as a one-to-one maximum matching guarantees each token is used once. It is a principled way to compare any two token sequences.
• I3. A tunable reward for contiguity. A single exponent lets the user decide how much word order should matter. A single number like this is handy when the target language has free word order, as Ukrainian does.
• I4. Grouping segments into pseudo-documents. Measuring correlation with humans at several aggregation sizes shows how reliable a metric is at different granularities. The same check is useful for any automatic metric meant for corpus-level or segment-level use.
• I5. Testing sensitivity to the number of references. Checking how a metric degrades with fewer references is practical for low-resource settings, where usually only one reference translation exists.