Elamur Research⌕
☰
home / papers / 2007 / (Meta-) Evaluation of Machine Translation

(Meta-) Evaluation of Machine Translation

Summary

This paper covers the translation shared task run as part of the ACL Workshop on Statistical Machine Translation in 2007. It ranks the submitted systems and also studies how machine translation should be evaluated. Systems translated between English and French, German, Spanish and Czech in both directions, giving 8 language pairs. Training data came from Europarl, News Commentary and CzEng, and a Moses-based baseline was supplied to all participants.

Volunteers and paid annotators carried out three kinds of manual judgment. The authors measured how long each judgment took and how consistent annotators were, both with themselves and with each other. Eleven automatic metrics were then compared against the human judgments using Spearman's ρ.

Two findings stand out. First, the classic fluency/adequacy scales gave weak inter-annotator agreement, while relative ranking gave much stronger agreement. Second, for translation into English, three metrics tracked human judgments better than Bleu: semantic role overlap, ParaEval-recall and Meteor. These results matter because they question the default practices of the field, both in how humans are asked to judge and in which metric is trusted.

Key points

  • The task drew 15 groups from 14 institutions, up from 14 groups the previous year. Most systems were phrase-based SMT, with some hybrid systems. The baseline used the open-source Moses decoder, which replaced Pharaoh.
  • Data:
    • Europarl supplied over 30 million training words per language, and News Commentary about 1 million.
    • Czech resources came from News Commentary and the CzEng Parallel Corpus.
    • There were two test sets: 2,000 Europarl sentences and a new 2,007-sentence News Commentary set built from Project Syndicate editorials, aligned across five languages.
  • Three manual evaluation types were used:
    • five-point fluency and adequacy scales, following the LDC/NIST convention;
    • ranking whole sentence translations against each other;
    • ranking translations of syntactic constituents taken from the parsed source sentence.
  • More than 100 people took part in manual evaluation, 75 of them for at least an hour. Total effort was 330 hours, compared with 180 the year before.
  • Into English, semantic role overlap, ParaEval-recall and Meteor beat Bleu. Out of English, Bleu and TER came closest to human judgments, but all correlations were clearly lower. Kappa agreement for fluency/adequacy was at most "fair", while both ranking modes agreed much better.

Interesting ideas

  • I1. Meta-evaluate the evaluation itself. Collect human judgments both to rank systems and to score the judging methods, using ease, agreement, repeatability and judgments per hour. Any annotation project could compare label schemes this way before committing to one.
  • I2. Check whether two label dimensions are really separate. Fluency and adequacy scores correlated highly, suggesting annotators cannot keep them apart. It is worth testing this for any multi-axis rating scheme, since a merged or ranking-based scheme may be cheaper and more reliable.
  • I3. Use relative ranking instead of absolute scales. Comparing outputs side by side produced better inter-annotator agreement than five-point scores. This is a useful default for human evaluation of generated text, including Ukrainian NLP outputs.
  • I4. Evaluate constituents, not just whole sentences. Parsing the source and using word alignments to highlight the matching target phrases lets annotators judge smaller, focused units. This could make manual checks of long or complex translations easier.
  • I5. Keep timing data clean and choose a robust correlation. Annotation sets lasting over 5 minutes were dropped as probably interrupted. Spearman's ρ was chosen because it suits ordinal data and makes fewer assumptions than Pearson. Both choices transfer easily to other annotation-cost and metric-validation studies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

206 papers of our library cite it (of the 140673 whose bibliographies we read); 30 of them have a page here, 176 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{callisonburch2007meta,
  title = {{(Meta-) Evaluation of Machine Translation}},
  author = {Callison-Burch, Chris and Fordyce, Cameron and Koehn, Philipp and Monz, Christof and Schroeder, Josh},
  booktitle = {Proceedings of the Second Workshop on Statistical Machine Translation},
  year = {2007},
  url = {https://aclanthology.org/W07-0718/}
}

References

52 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Enrique Amigó, Jesús Giménez, Julio Gonzalo, and Lluı́s Màrquez. 2006. MT Evaluation: Human-Like vs. Human Acceptable. In Proceedings of COLING-ACL06. find
  2. Abhishek Arun and Frank Keller. 2005. Lexicalization in crosslinguistic probabilistic parsing: The case of French. In Proceedings of ACL. find
  3. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for MT evaluation with improved correlation with human judgments. In Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, Ann Arbor, Michigan. find our card
  4. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In ACL-2005. find our card
  5. Dan Bikel. 2002. Design of a multi-lingual, parallel-processing statistical parsing engine. In Proceedings of HLT. find
  6. John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2003. Confidence estimation for machine translation. CLSP Summer Workshop Final Report WS2003, Johns Hopkins University. find
  7. Ondřej Bojar and Zdeněk Žabokrtský. 2006. CzEng: Czech-English Parallel Corpus, Release version 0.5. Prague Bulletin of Mathematical Linguistics, 86. find
  8. Ondřej Bojar. 2004. Problems of inducing large coverage constraint-based dependency grammar for Czech. In Constraint Solving and Language Processing, CSLP 2004, volume LNAI 3438. Springer. find
  9. Ondřej Bojar. 2007. English-to-Czech factored machine translation. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find
  10. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of Bleu in machine translation research. In Proceedings of EACL. find
  11. Chris Callison-Burch. 2007. Paraphrasing and Translation. Ph.D. thesis, University of Edinburgh, Scotland. find
  12. Jean Carletta. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249–254. ↗ our card
  13. Yu Chen, Andreas Eisele, Christian Federmann, Eva Hasler, Michael Jellinghaus, and Silke Theison. 2007. Multi-engine machine translation with an open-source decoder for statistical machine translation. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find
  14. Jorge Civera and Alfons Juan. 2007. Domain adaptation in statistical machine translation with mixture modelling. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find
  15. Marta R. Costa-Jussà and José A.R. Fonollosa. 2007. Analysis of statistical and morphological classes to generate weighted reordering hypotheses on a statistical machine translation system. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find
  16. Deborah Coughlin. 2003. Correlating automated and human assessments of machine translation quality. In Proceedings of MT Summit IX. find
  17. Brooke Cowan and Michael Collins. 2005. Morphology and reranking for the statistical parsing of Spanish. In Proceedings of EMNLP 2005. find
  18. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Human Language Technology: Notebook Proceedings, pages 128–132, San Diego. find
  19. Amit Dubey. 2005. What to do when lexicalization fails: parsing German with suffix analysis and smoothing. In Proceedings of ACL. find
  20. Loı̈c Dugast, Jean Senellart, and Philipp Koehn. 2007. Statistical post-editing on SYSTRAN’s rule-based translation system. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find
  21. Christopher J. Dyer. 2007. The ‘noisier channel’: translation from morphologically complex languages. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find
  22. Matthias Eck and Chiori Hori. 2005. Overview of the IWSLT 2005 evaluation campaign. In Proceedings of International Workshop on Spoken Language Translation. find
  23. Jesús Giménez and Lluı́s Màrquez. 2007. Linguistic features for automatic evaluation of heterogenous mt systems. In Proceedings of ACL Workshop on Statistical Machine Translation. find
  24. Maria Holmqvist, Sara Stymne, and Lars Ahrenberg. 2007. Getting to know Moses: Initial experiments on German-English factored translation. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find
  25. Douglas Jones, Wade Shen, Neil Granoien, Martha Herzog, and Clifford Weinstein. 2005. Measuring translation quality by testing english speakers with a new defense language proficiency test for arabic. In Proceedings of the 2005 International Conference on Intelligence Analysis. find
  26. Philipp Koehn and Christof Monz. 2005. Shared task: Statistical machine translation between European languages. In Proceedings of ACL 2005 Workshop on Parallel Text Translation. find
  27. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings of NAACL 2006 Workshop on Statistical Machine Translation. find our card
  28. Philipp Koehn and Josh Schroeder. 2007. Experiments in domain adaptation for statistical machine translation. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find our card
  29. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of HLT/NAACL. ↗
  30. Philipp Koehn, Nicola Bertoldi, Ondrej Bojar, Chris Callison-Burch, Alexandra Constantin, Brooke Cowan, Chris Dyer, Marcello Federico, Evan Herbst, Hieu Hoang, Christine Moran, Wade Shen, and Richard Zens. 2006. Factored translation models. CLSP Summer Workshop Final Report WS-2006, Johns Hopkins University. find our card
  31. Philipp Koehn. 2004. Pharaoh: A beam search decoder for phrase-based statistical machine translation models. In Proceedings of AMTA. find
  32. Philipp Koehn. 2005. A parallel corpus for statistical machine translation. In Proceedings of MT-Summit. find
  33. J. R. Landis and G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159–174. ↗
  34. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Workshop on Statistical Machine Translation, Prague, June. Association for Computational Linguistics. find our card
  35. LDC. 2005. Linguistic data annotation specification: Assessment of fluency and adequacy in translations. Revision 1.5. find
  36. Audrey Lee. 2006. NIST 2006 machine translation evaluation official results. Official release of automatic evaluation scores for all submissions, November. find
  37. Ding Liu and Daniel Gildea. 2007. Source-language features and maximum correlation training for machine translation evaluation. In Proceedings of NAACL. find
  38. Dan Melamed, Ryan Green, and Jospeh P. Turian. 2003. Precision and recall of machine translation. In Proceedings of HLT/NAACL. find
  39. Preslav Nakov and Marti Hearst. 2007. UCB system description for the WMT 2007 shared task. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find
  40. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51, March. ↗
  41. Franz Josef Och and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computational Linguistics. ↗
  42. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of ACL. find our card
  43. Michael Paul. 2006. Overview of the IWSLT 2006 evaluation campaign. In Proceedings of International Workshop on Spoken Language Translation. find
  44. Matthias Paulik, Kay Rottmann, Jan Niehues, Silja Hildebrand, and Stephan Vogel. 2007. The ISL phrase-based MT system for the 2007 ACL Workshop on Statistical Machine Translation. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find
  45. Maja Popovic and Hermann Ney. 2007. Word error rates: Decomposition over POS classes and applications for error analysis. In Proceedings of ACL Workshop on Statistical Machine Translation. find
  46. Mark Przybocki. 2004. NIST 2004 machine translation evaluation results. Confidential e-mail to workshop participants, May. find
  47. Holger Schwenk. 2007. Building a statistical machine translation system for French using the Europarl corpus. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find
  48. Michel Simard, Nicola Ueffing, Pierre Isabelle, and Roland Kuhn. 2007. Rule-based translation with statistical phrase-based post-editing. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find
  49. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Statistical Machine Translation in the Americas. find our card
  50. Nicola Ueffing, Michel Simard, Samuel Larkin, and Howard Johnson. 2007. NRC’s PORTAGE system for WMT 2007. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. find
  51. Liang Zhou, Chin-Yew Lin, and Eduard Hovy. 2006. Re-evaluating machine translation results with paraphrase support. In Proceedings of EMNLP. find
  52. Andreas Zollmann, Ashish Venugopal, Matthias Paulik, and Stephan Vogel. 2007. The syntax augmented MT (SAMT) system for the shared task in the 2007 ACL Workshop on Statistical Machine Translation. In Proceedings of the ACL-2007 Workshop on Statistical Machine Translation (WMT-07), Prague. CONSTITUENT ADEQUACY FLUENCY RANK system German-English Europarl 0.511 0.496 0.395 0.541 0.55 0.415 0.474 0.459 0.354 0.334 0.404 0.119 0.562 0.594 0.530 0.53 0.554 0.43 0.534 0.533 0.384 cmu-uka liu nrc saar systran uedin upc 0.206 0.234 0.214 0.104 0.302 0.187 0.214 German-English News Corpus nrc 0.459 0.429 0.325 0.245 saar 0.278 0.341 0.108 0.125 systran 0.552 0.56 0.563 0.344 uedin 0.508 0.536 0.485 0.332 upc 0.536 0.512 0.476 0.330 English-German Europarl 0.557 0.508 0.416 0.534 0.511 0.328 0.369 0.383 0.172 0.543 0.525 0.511 0.569 0.576 0.389 0.565 0.522 0.438 cmu-uka nrc saar systran uedin upc 0.333 0.321 0.196 0.295 0.350 0.3 English-German News Corpus nrc 0.453 0.4 0.437 0.340 saar 0.186 0.273 0.108 0.121 systran 0.542 0.556 0.582 0.351 ucb 0.415 0.403 0.332 0.289 uedin 0.472 0.445 0.455 0.303 upc 0.505 0.475 0.377 0.349 Table 9: Human evaluation for German-English submissions CONSTITUENT ADEQUACY FLUENCY RANK system Spanish-English Europarl cmu-syntax 0.552 0.568 0.478 cmu-uka 0.557 0.564 0.392 nrc 0.477 0.489 0.382 saar 0.328 0.336 0.126 systran 0.525 0.566 0.453 uedin 0.593 0.610 0.419 upc 0.587 0.604 0.5 upv 0.562 0.573 0.326 0.152 0.139 0.143 0.075 0.156 0.14 0.188 0.154 find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/burch-2007-meta-evaluation-machine-translation/