📄 Validation of an Automatic Metric for the Accuracy of Pronoun Translation (APT)

Summary

Scoring pronoun translation in MT is difficult because the correct output depends on how the antecedent was translated and on non-referential uses. For this reason, the DiscoMT 2015 shared task used human gap-filling judgments, which are expensive and cannot be reused for new outputs. The authors propose APT, a reference-based automatic score for English it/they translated into French.

APT works in three steps:

• It word-aligns source, reference and candidate with GIZA++ (merged via Moses' grow-diag-final).
• It repairs pronoun alignments with heuristics that search the target window around neighbouring words' alignments.
• It places each source pronoun into one of six cases: identical, equivalent, different, absent in candidate, absent in reference, or absent in both. Weighted case counts give the share of correctly translated pronouns.

On the seven DiscoMT 2015 systems, APT correlates with human scores at 0.993–0.999 Pearson and up to 1.000 Spearman. This beats BLEU, METEOR and the pronoun-specific DiscoMT metrics, showing that cheap, repeatable reference-based pronoun evaluation is viable.

Key points

• Validation data: DiscoMT 2015 English→French task with 2,093 source sentences. Humans judged a sample of 210 pronouns for each of six submitted systems plus a baseline.
• Alignment check on 100 WIT3 (TED Talks) sentences: GIZA++ missed 19 pronoun alignments and got 4 wrong. The heuristics fixed 22 of these 23 errors.
• Comparison metrics (BLEU, METEOR, AutoP/AutoR/AutoF) reached only 0.972–0.986 Pearson.
• Weight tuning for "equivalent" pronouns (case 2) and both-missing pronouns (case 6) changed correlations only marginally. With heuristic alignment, w₂ = 0.5 performed slightly best.
• Error analysis: 84% of identical-to-reference pronouns were judged correct by humans. However, 26% of "different" ones and 53% of reference-missing ones were also correct, revealing legitimate variation the metric cannot capture.
• The code is released at github.com/idiap/APT.