Elamur Research
☰
home / papers / 2010 / Automatic Evaluation of Translation Quality for Distant Language Pairs

Automatic Evaluation of Translation Quality for Distant Language Pairs

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/D10-1092.pdf

Summary

Standard MT metrics such as BLEU, NIST, PER and TER rely on local n-gram overlap or tolerate reordering. They therefore barely penalize large word-order errors, which are common when SMT systems translate between structurally different languages like Japanese (SOV) and English (SVO). An output that swaps cause and effect still gets a BLEU of 0.740. The authors propose scoring a hypothesis by how well it preserves the reference word order. Words are aligned through unique unigrams, with unique bigrams as a fallback. Kendall's τ or Spearman's ρ is then computed over the aligned positions and normalized to [0, 1] (NKT, NSR). Because unaligned words would otherwise inflate the score, the authors multiply the score by unigram precision raised to a small power, replacing BLEU's Brevity Penalty. Meta-evaluation against human adequacy and fluency judgments on NTCIR-7 PATMT JE data shows the precision-modified metrics beat conventional ones. On WMT-07 European pairs they perform about as well as existing metrics. This makes the approach useful for tuning and evaluating MT between distant languages.

Key points

  • Alignment and correlation: The alignment algorithm (Figure 1) uses only unigrams and bigrams that occur exactly once in each sentence. On the swapped "because" example, it yields τ = −0.236 and ρ = −0.591, compared with BLEU's 0.740.
  • Metric variants: The tested variants include NKT and NSR, plus precision-weighted versions with exponents 1/8, 1/4 and 1/2 (e.g., NKTP, NSRP). On NTCIR-7, τ was slightly better than ρ, and precision improved ρ.
  • Penalty choice: BLEU's Brevity Penalty proved harmful for these metrics, while the precision-based penalty worked well.
  • WMT-07 results: The metrics were comparable to BLEU, WER and ROUGE overall on WMT-07 "rank" judgments. ROUGE-L and ROUGE-S did better on French News Corpus and German Europarl.
  • Comparison with related work: Birch et al. (2010) independently proposed a Kendall's τ metric with a square-root transform. On NTCIR-7, that transform raised Pearson correlation with adequacy (0.922 → 0.941) but lowered the other correlation scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

185 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 183 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{isozaki2010automatic,
  title = {{Automatic Evaluation of Translation Quality for Distant Language Pairs}},
  author = {Isozaki, Hideki and Hirao, Tsutomu and Duh, Kevin and Sudoh, Katsuhito and Tsukada, Hajime},
  booktitle = {Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://aclanthology.org/D10-1092/}
}

References

16 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for MT evaluation with improved correlation with human judgements. In Proc. of ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and Summarization, pages 65–72. find
  2. Alexandra Birch and Miles Osborne. 2010. LRscore for evaluating lexical and reordering quality in MT. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 327– 332. find
  3. Alexandra Birch, Miles Osborne, and Phil Blunsom. 2010. Metrics for MT evaluation: evaluating reordering. Machine Translation, 24(1):15–26. find
  4. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluatiing the role of Bleu in machine translation research. In Proc. of the Conference of the European Chapter of the Association for Computational Linguistics, pages 249–256. find
  5. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Chrstof Monz, and Josh Schroeder. 2007. (Meta-)Evaluation of machine translation. In Proc. of the Workshop on Machine Translation (WMT), pages 136–158. find
  6. Etienne Denoual and Yves Lepage. 2005. BLEU in characters: towards automatic MT evaluation in languages without word delimiters. In Companion Volume to the Proceedings of the Second International Joint Conference on Natural Language Processing, pages 81–86. find
  7. Hiroshi Echizen-ya and Kenji Araki. 2007. Automatic evaluation of machine translation based on recursive acquisition of an intuitive common parts continuum. In Proceedings of MT Summit XII Workshop on Patent Translation, pages 151–158. find
  8. Hiroshi Echizen-ya, Terumasa Ehara, Sayori Shimohata, Atsushi Fujii, Masao Utiyama, Mikio Yamamoto, Takehito Utsuro, and Noriko Kando. 2009. Meta-evaluation of automatic evaluation methods for machine translation using patent translation data in ntcir-7. In Proceedings of the 3rd Workshop on Patent Translation, pages 9–16. find
  9. Atsushi Fujii, Masao Utiyama, Mikio Yamamoto, and Takehito Utsuro. 2008. Overview of the patent translation task at the NTCIR-7 workshop. In Working Notes of the NTCIR Workshop Meeting (NTCIR), pages 389–400. find
  10. Hideki Isozaki, Katsuhito Sudoh, Hajime Tsukada, and Kevin Duh. 2010. Head Finalization: A simple reordering rule for SOV languages. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 250–257. find
  11. Maurice G. Kendall. 1975. Rank Correlation Methods. Charles Griffin. find
  12. Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proc. of the North American Chapter of the Association of Computational Linguistics (NAACL), pages 71–78. find
  13. Dan Melamed, Ryan Green, and Joseph P. Turian. 2007. Precision and recall of machine translation. In Proc. of NAACL-HLT, pages 61–63. find
  14. Kishore Papineni, Salim Roukos, Todd Ward, John Henderson, and Florence Reeder. 2002a. Corpus-based comprehensive and diagnostic MT evaluation: Initial Arabic, Chinese, French, and Spanish Results. In Proc. of the International Conference on Human Language Technology Research (HLT), pages 132–136. find
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002b. BLEU: a method for automatic evaluation of machine translation. In Proc. of the Annual Meeting of the Association of Computational Linguistics (ACL), pages 311–318. find our card
  16. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Machine Translation in the Americas. find our card