Elamur Research⌕
☰
home / papers / 2006 / Manual and Automatic Evaluation of Machine Translation between European Languages

Manual and Automatic Evaluation of Machine Translation between European Languages

Summary

This paper reports on the shared task of the 2006 NAACL/HLT Workshop on Statistical Machine Translation. It asks how well current MT systems handle European languages and how automatic and human judgements of quality relate. Six directions were tested: French, German and Spanish into English and from English. Training used the Europarl corpus. The organizers gave participants a ready-made baseline, including the Pharaoh decoder, trained language models and a training script, to make entry easier. Systems were tested on an in-domain Europarl set and a new out-of-domain set of political commentary. Scoring combined BLEU, with significance testing, and volunteer ratings of fluency and adequacy. Most of the 14 submissions were phrase-based statistical systems, and their performance was largely indistinguishable. The commercial rule-based Systran system was undervalued by BLEU on in-domain data, but this gap mostly vanished out of domain. The work matters because it set public baselines and released human annotations for future metric research. It also documents practical weaknesses of graded manual scoring.

Key points

  • 14 teams from 11 institutions took part. Finnish, used in the previous year's task, was dropped, partly because finding enough Finnish judges seemed unlikely.
  • The test sets were:
    • In-domain: 2,000 Europarl sentences from the last quarter of 2000, held out from training.
    • Out-of-domain: 1,064 sentences aligned across four languages, taken from 29 Project Syndicate editorials.
  • BLEU was computed against a single reference, after the organizers retokenized and lowercased all outputs themselves.
  • Significance testing used two methods:
    • Bootstrap resampling (1,000 samples) to get confidence intervals.
    • A sign test over 20-sentence blocks (100 blocks in-domain, 53 out-of-domain) to compare pairs of systems.
  • About 180 hours of volunteer work produced fluency and adequacy ratings on a 1–5 scale.
  • Average BLEU ranged from 28.33 for English-French (easiest) to 14.01 for English-German (hardest). Statistical systems trailed Systran when translating into morphologically rich languages.

Interesting ideas

  • I1. Ship a full baseline kit with a shared task. Providing data, language models, a decoder and training scripts lets newcomers compete and gives every result a common reference point. This would help any Ukrainian NLP benchmark attract participants.
  • I2. Test significance on blocks, not just the whole corpus. Splitting the test set into small chunks and applying a sign test is a cautious alternative to bootstrap resampling, which can be too optimistic. It works for any metric that is only meaningful at corpus level.
  • I3. Always add an out-of-domain test set. The comparison showed that metric bias can depend on domain. Checking a model on a second text type can therefore reveal whether a metric's preferences are stable.
  • I4. Prefer ranking over absolute scales. Judges found it hard to apply 1–5 scores consistently, and judge differences distorted comparisons across language pairs. Relative ranking of several outputs side by side is likely to give more reliable human evaluation.
  • I5. Make human evaluation feasible for judges. Useful design choices include:
    • splitting long sentences into clauses for judging;
    • letting judges skip items they cannot assess;
    • checking the quality of reference translations, which annotators estimated was faulty for up to about 10% of sentences;
    • providing multiple references where possible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

164 papers of our library cite it (of the 140673 whose bibliographies we read); 31 of them have a page here, 133 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{koehn2006manual,
  title = {{Manual and Automatic Evaluation of Machine Translation between European Languages}},
  author = {Koehn, Philipp and Monz, Christof},
  booktitle = {Proceedings on the Workshop on Statistical Machine Translation},
  year = {2006},
  doi = {10.3115/1654650.1654666},
  url = {https://doi.org/10.3115/1654650.1654666}
}

References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Birch, A., Callison-Burch, C., Osborne, M., and Koehn, P. (2006). Constraining the phrase-based, joint probability statistical translation model. In Proceedings on the Workshop on Statistical Machine Translation, pages 154–157, New York City. Association for Computational Linguistics. find
  2. Callison-Burch, C., Osborne, M., and Koehn, P. (2006). Re-evaluating the role of BLEU in machine translation research. In Proceedings of EACL. find
  3. Collins, M., Koehn, P., and Kucerova, I. (2005). Clause restructuring for statistical machine translation. In Proceedings of ACL. find our card
  4. Costa-jussà, M. R., Crego, J. M., de Gispert, A., Lambert, P., Khalilov, M., Mariño, J. B., Fonollosa, J. A. R., and Banchs, R. (2006). Talp phrase-based statistical translation system for european language pairs. In Proceedings on the Workshop on Statistical Machine Translation, pages 142– 145, New York City. Association for Computational Linguistics. find
  5. Crego, J. M., de Gispert, A., Lambert, P., Costa-jussà, M. R., Khalilov, M., Banchs, R., Mariño, J. B., and Fonollosa, J. A. R. (2006). N-gram-based smt system enhanced with reordering patterns. In Proceedings on the Workshop on Statistical Machine Translation, pages 162–165, New York City. Association for Computational Linguistics. find
  6. Doddington, G. (2002). The NIST automated measure and its relation to IBM’s BLEU. In Proceedings of LREC-2002 Workshopon Machine Translation Evaluation: Human Evaluators Meet Automated Metrics, Gran Canaria, Spain. find
  7. Eck, M. and Hori, C. (2005). Overview of the iwslt 2005 evaluation campaign. In Proc. of the International Workshop on Spoken Language Translation. find
  8. Giménez, J. and Màrquez, L. (2006). The ldv-combo system for smt. In Proceedings on the Workshop on Statistical Machine Translation, pages 166–169, New York City. Association for Computational Linguistics. find
  9. Johnson, H., Sadat, F., Foster, G., Kuhn, R., Simard, M., Joanis, E., and Larkin, S. (2006). Portage: with smoothed phrase tables and segment choice models. In Proceedings on the Workshop on Statistical Machine Translation, pages 134–137, New York City. Association for Computational Linguistics. find
  10. Koehn, P. (2004). Statistical significance tests for machine translation evaluation. In Lin, D. and Wu, D., editors, Proceedings of EMNLP 2004, pages 388–395, Barcelona, Spain. Association for Computational Linguistics. find
  11. Koehn, P. and Monz, C. (2005). Shared task: Statistical machine translation between European languages. In Proceedings of the ACL Workshop on Building and Using Parallel Texts, pages 119– 124, Ann Arbor, Michigan. Association for Computational Linguistics. find
  12. Li, A. (2005). Results of the 2005 NIST machine translation evaluation. In Machine Translation Workshop. find
  13. Menezes, A., Toutanova, K., and Quirk, C. (2006). Microsoft research treelet translation system: Naacl 2006 europarl evaluation. In Proceedings on the Workshop on Statistical Machine Translation, pages 158–161, New York City. Association for Computational Linguistics. find
  14. Olteanu, M., Davis, C., Volosen, I., and Moldovan, D. (2006a). Phramer - an open source statistical phrase-based translator. In Proceedings on the Workshop on Statistical Machine Translation, pages 146–149, New York City. Association for Computational Linguistics. find
  15. Olteanu, M., Suriyentrakorn, P., and Moldovan, D. (2006b). Language models and reranking for machine translation. In Proceedings on the Workshop on Statistical Machine Translation, pages 150– 153, New York City. Association for Computational Linguistics. find
  16. Patry, A., Gotti, F., and Langlais, P. (2006). Mood at work: Ramses versus pharaoh. In Proceedings on the Workshop on Statistical Machine Translation, pages 126–129, New York City. Association for Computational Linguistics. find
  17. Przybocki, M. (2004). NIST machine translation 2004 evaluation – summary of results. In Machine Translation Evaluation Workshop. find
  18. Riezler, S. and Maxwell, J. T. (2005). On some pitfalls in automatic evaluation and significance testing for MT. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 57–64, Ann Arbor, Michigan. Association for Computational Linguistics. find
  19. Sánchez, J. A. and Benedı́, J. M. (2006). Stochastic inversion transduction grammars for obtaining word phrases for phrase-based statistical machine translation. In Proceedings on the Workshop on Statistical Machine Translation, pages 130–133, New York City. Association for Computational Linguistics. find
  20. Watanabe, T., Tsukada, H., and Isozaki, H. (2006). Ntt system description for the wmt2006 shared task. In Proceedings on the Workshop on Statistical Machine Translation, pages 122–125, New York City. Association for Computational Linguistics. find
  21. Zollmann, A. and Venugopal, A. (2006). Syntax augmented machine translation via chart parsing. In Proceedings on the Workshop on Statistical Machine Translation, pages 138–141, New York City. Association for Computational Linguistics. French-English (In Domain) Adequacy (rank) +0.19±0.08 (1-7) +0.14±0.07 (1-6) +0.13±0.08 (1-7) +0.13±0.08 (1-8) +0.12±0.10 (1-7) +0.11±0.08 (1-8) +0.10±0.08 (3-7) -0.02±0.08 (5-8) -0.08±0.09 (9) -0.76±0.09 (10) upc-jmc lcc utd upc-mr nrc ntt cmu rali systran upv Spanish-English (In Domain) Adequacy (rank) +0.15±0.08 (1-7) +0.10±0.08 (1-7) +0.08±0.07 (1-8) +0.08±0.06 (1-8) +0.06±0.10 (2-8) +0.06±0.07 (1-8) +0.03±0.11 (1-8) +0.00±0.07 (3-9) -0.10±0.07 (7-9) -0.45±0.10 (10) upc-jmc ntt lcc utd nrc upc-mr uedin-birch rali upc-jg upv German-English (In Domain) Adequacy (rank) +0.30±0.09 (1-2) +0.15±0.07 (2-7) +0.12±0.07 (2-7) +0.08±0.07 (3-7) +0.07±0.08 (2-9) +0.00±0.09 (3-9) -0.01±0.06 (4-9) -0.02±0.09 (2-9) -0.05±0.10 (3-9) -0.55±0.09 (10) uedin-phi lcc nrc utd ntt upc-mr rali upc-jmc systran upv Figure 7: Evaluation of translation to English on in-domain test data Fluency (rank) +0.09±0.08 (1-8) +0.13±0.06 (1-7) +0.14±0.07 (1-6) +0.13±0.07 (1-6) +0.06±0.11 (2-6) +0.14±0.08 (2-8) +0.05±0.07 (4-8) +0.00±0.08 (3-9) -0.17±0.09 (8-9) -0.52±0.09 (10) BLEU (rank) 30.42±0.86 (1-6) 30.81±0.85 (1-4) 30.53±0.87 (2-7) 30.33±0.88 (1-7) 29.62±0.84 (8) 30.72±0.87 (1-7) 30.18±0.80 (2-7) 30.39±0.91 (3-7) 21.44±0.65 (10) 24.10±0.89 (9) Fluency (rank) +0.18±0.08 (1-6) +0.10±0.08 (1-8) +0.04±0.06 (2-8) +0.08±0.07 (2-7) +0.08±0.07 (1-9) +0.08±0.07 (1-6) -0.07±0.15 (2-10) -0.02±0.07 (3-9) -0.11±0.07 (6-9) -0.41±0.10 (9-10) BLEU (rank) 31.01±0.97 (1-5) 31.29±0.88 (1-5) 31.46±0.87 (1-4) 31.10±0.89 (1-5) 30.04±0.79 (6) 29.43±0.83 (7) 29.01±0.81 (8) 30.80±0.87 (2-5) 28.03±0.83 (9) 23.91±0.83 (10) Fluency (rank) +0.33±0.08 (1) +0.12±0.07 (2-7) +0.14±0.07 (2-6) +0.01±0.08 (2-8) +0.06±0.09 (2-8) -0.21±0.09 (6-9) +0.00±0.07 (3-9) -0.04±0.09 (3-9) -0.05±0.09 (3-9) -0.38±0.08 (10) BLEU (rank) 27.30±0.86 (1) 25.97±0.81 (2) 24.54±0.80 (5-7) 25.44±0.85 (3-4) 25.64±0.83 (3-4) 23.68±0.79 (8) 24.60±0.80 (5-7) 24.43±0.86 (5-7) 15.86±0.59 (10) 18.08±0.77 (9) English-French (In Domain) Adequacy (rank) +0.08±0.09 (1-5) +0.08±0.08 (1-4) +0.03±0.09 (1-6) -0.01±0.12 (2-7) -0.03±0.07 (3-7) -0.08±0.09 (1-7) -0.09±0.09 (4-7) nrc upc-mr upc-jmc systran utd rali ntt English-Spanish (In Domain) Adequacy (rank) +0.23±0.09 (1-5) +0.20±0.09 (1-4) +0.18±0.08 (1-5) +0.12±0.09 (2-7) +0.10±0.09 (3-7) +0.04±0.10 (2-7) -0.05±0.08 (5-8) -0.18±0.14 (6-9) -0.32±0.11 (9) -0.83±0.15 (9-10) ms upc-mr utd nrc ntt upc-jmc rali uedin-birch upc-jg upv English-German (In Domain) Adequacy (rank) +0.28±0.08 (1-3) +0.19±0.08 (1-5) +0.17±0.08 (1-5) +0.17±0.08 (2-4) +0.08±0.10 (3-6) -0.08±0.11 (5-6) -0.84±0.12 (7) upc-mr ntt upc-jmc nrc rali systran upv Figure 8: Evaluation of translation from English on in-domain test data Fluency (rank) +0.09±0.09 (1-5) +0.04±0.07 (1-5) +0.02±0.08 (1-6) +0.06±0.12 (1-6) -0.05±0.07 (3-7) -0.09±0.09 (2-7) -0.06±0.08 (4-7) BLEU (rank) 31.75±0.83 (1-6) 31.50±0.76 (1-6) 31.75±0.78 (1-5) 25.07±0.71 (7) 31.42±0.85 (3-6) 31.79±0.85 (1-6) 31.92±0.84 (1-5) Fluency (rank) +0.13±0.09 (1-7) +0.17±0.09 (1-5) +0.15±0.08 (1-6) +0.17±0.08 (1-6) +0.14±0.08 (1-6) +0.01±0.08 (2-7) -0.03±0.08 (6-8) -0.17±0.13 (6-10) -0.37±0.09 (8-10) -0.59±0.15 (8-10) BLEU (rank) 29.76±0.82 (7-8) 31.06±0.86 (1-4) 30.73±0.90 (1-4) 29.97±0.86 (5-6) 30.93±0.85 (1-4) 30.44±0.86 (1-4) 29.38±0.85 (5-6) 28.49±0.87 (7-8) 27.46±0.78 (9) 23.17±0.73 (10) Fluency (rank) +0.14±0.08 (1-5) +0.09±0.06 (2-6) +0.13±0.08 (1-4) +0.11±0.08 (1-5) +0.03±0.09 (2-6) +0.00±0.10 (3-6) -0.51±0.10 (7) BLEU (rank) 17.24±0.81 (3-5) 18.15±0.89 (1-3) 17.73±0.81 (1-3) 17.52±0.78 (4-5) 17.93±0.85 (1-4) 9.84±0.52 (7) 13.37±0.78 (6) French-English (Out of Domain) Adequacy (rank) +0.23±0.09 (1-5) +0.22±0.11 (1-8) +0.19±0.15 (1-8) +0.13±0.12 (1-9) +0.12±0.12 (2-8) +0.04±0.10 (1-9) -0.02±0.12 (3-9) -0.03±0.14 (3-8) -0.09±0.12 (4-9) -0.76±0.16 (10) upc-jmc cmu systran lcc upc-mr utd ntt nrc rali upv Spanish-English (Out of Domain) Adequacy (rank) +0.28±0.10 (1-2) +0.25±0.16 (1-7) +0.18±0.16 (2-8) +0.11±0.10 (2-7) +0.08±0.11 (2-8) +0.04±0.10 (4-9) +0.03±0.11 (2-9) -0.09±0.11 (4-9) -0.09±0.11 (4-9) -0.63±0.14 (10) upc-jmc uedin-birch nrc ntt upc-mr lcc utd upc-jg rali upv German-English (Out of Domain) Adequacy (rank) +0.30±0.12 (1-4) +0.22±0.09 (1-6) +0.18±0.10 (1-6) +0.08±0.09 (2-7) +0.07±0.12 (1-9) +0.04±0.10 (3-8) +0.02±0.10 (4-8) -0.01±0.10 (4-8) -0.14±0.08 (8-9) -0.64±0.11 (10) systran uedin-phi lcc utd ntt nrc upc-mr upc-jmc rali upv Figure 9: Evaluation of translation to English on out-of-domain test data Fluency (rank) +0.13±0.11 (1-8) +0.13±0.09 (1-9) +0.15±0.14 (1-7) +0.11±0.11 (1-9) +0.11±0.10 (1-7) +0.01±0.10 (1-8) +0.08±0.11 (1-9) +0.00±0.11 (3-9) -0.10±0.11 (5-9) -0.58±0.14 (10) BLEU (rank) 21.79±0.92 (1-4) 21.15±0.86 (4-7) 19.42±0.82 (9) 21.77±0.88 (1-5) 21.95±0.94 (1-3) 21.39±0.94 (3-7) 21.34±0.85 (3-7) 21.15±0.86 (3-7) 20.17±0.85 (8) 15.55±0.79 (10) Fluency (rank) +0.17±0.10 (1-6) +0.18±0.19 (1-6) +0.09±0.09 (1-8) +0.17±0.10 (2-6) +0.10±0.10 (1-7) +0.07±0.11 (3-7) +0.03±0.10 (2-8) -0.09±0.09 (7-9) -0.15±0.11 (6-9) -0.47±0.11 (10) BLEU (rank) 27.92±0.94 (1-3) 25.20±0.91 (5-8) 25.40±0.94 (5-7) 26.85±0.89 (3-4) 25.62±0.87 (5-8) 27.18±0.92 (1-4) 27.41±0.96 (1-3) 23.42±0.87 (9) 25.03±0.91 (6-8) 19.17±0.78 (10) Fluency (rank) +0.21±0.12 (1-4) +0.21±0.10 (1-7) +0.20±0.10 (1-7) +0.07±0.08 (2-6) +0.21±0.13 (1-7) +0.04±0.09 (2-8) -0.11±0.09 (6-8) -0.04±0.11 (3-9) -0.14±0.08 (8-9) -0.54±0.09 (10) BLEU (rank) 15.56±0.71 (7-9) 18.87±0.84 (1) 17.96±0.79 (2-3) 16.97±0.76 (4-6) 17.37±0.76 (3-5) 15.93±0.76 (7-8) 16.89±0.79 (4-6) 17.57±0.80 (2-5) 15.22±0.69 (8-9) 11.78±0.71 (10) English-French (Out of Domain) Adequacy (rank) +0.50±0.20 (1) +0.09±0.11 (2-5) +0.09±0.11 (2-4) -0.02±0.11 (2-6) -0.12±0.12 (4-7) -0.13±0.13 (4-7) -0.23±0.12 (4-7) systran upc-jmc upc-mr utd rali nrc ntt English-Spanish (Out of Domain) Adequacy (rank) +0.35±0.11 (1-3) +0.33±0.16 (1-7) +0.21±0.13 (2-6) +0.18±0.12 (1-6) +0.17±0.15 (2-7) +0.12±0.13 (2-7) -0.17±0.16 (6-8) -0.36±0.24 (6-10) -0.45±0.13 (8-9) -1.09±0.21 (9) upc-mr ms utd nrc upc-jmc ntt rali uedin-birch upc-jg upv English-German (Out of Domain) Adequacy (rank) +0.47±0.15 (1) +0.31±0.13 (2-3) +0.22±0.14 (2-3) +0.13±0.12 (4-6) +0.00±0.11 (4-6) -0.03±0.12 (4-6) -0.94±0.13 (7) systran upc-mr upc-jmc rali nrc ntt upv Figure 10: Evaluation of translation from English on out-of-domain test data Fluency (rank) +0.41±0.18 (1) +0.09±0.11 (2-4) +0.04±0.09 (2-4) -0.05±0.09 (2-6) -0.17±0.12 (5-7) -0.16±0.10 (4-7) -0.06±0.10 (4-7) BLEU (rank) 25.31±0.88 (1) 23.30±0.75 (2-6) 23.21±0.75 (2-6) 22.79±0.86 (7) 23.34±0.89 (2-6) 23.66±0.91 (2-5) 22.99±0.96 (3-6) Fluency (rank) +0.19±0.10 (1-6) +0.15±0.13 (1-8) +0.13±0.11 (1-7) +0.07±0.11 (2-7) +0.24±0.12 (1-6) +0.12±0.13 (1-7) -0.05±0.13 (4-8) -0.16±0.16 (5-9) -0.42±0.10 (9-10) -0.64±0.19 (8-9) BLEU (rank) 26.62±0.92 (1-2) 26.15±0.88 (6-7) 25.26±0.78 (3-5) 25.58±0.85 (3-5) 25.59±0.95 (3-5) 26.52±0.90 (1-2) 24.03±0.83 (6-8) 23.18±0.88 (7-8) 22.04±0.84 (9) 16.83±0.72 (10) Fluency (rank) +0.39±0.15 (1-2) +0.21±0.11 (1-3) +0.01±0.10 (3-6) -0.06±0.10 (4-6) +0.05±0.09 (2-6) +0.08±0.11 (3-5) -0.57±0.10 (7) BLEU (rank) 10.78±0.69 (1-6) 10.96±0.70 (1-5) 10.64±0.66 (1-6) 10.57±0.65 (1-6) 10.64±0.65 (2-6) 10.51±0.64 (1-6) 6.55±0.53 (7) French-English In domain Adequacy 0.3 0.2 0.1 -0.0 •systran -0.1 -0.2 -0.3 -0.4 -0.5 -0.6 -0.7 •upv -0.8 upc-jmc utd upc-mr & • . nrc• • • • • •lcc % - ntt cmu • rali Fluency 0.2 0.1 -0.0 -0.1 •systran -0.2 -0.3 -0.4 •upv -0.5 upc-mr & . utd • • •ntt • - upc-jmc• • •cmu lcc % nrc • rali Figure 11: Correlation between manual and automatic scores for French-English Out of Domain Adequacy 0.3 systran• cmu• •upc-jmc 0.2 lcc & •upc-mr 0.1 utd• • nrc• -0.0 •ntt -0.1 rali• -0.2 -0.3 -0.4 -0.5 -0.6 -0.7 •upv B LEU -0.8 B LEU Fluency 0.2 . upc-jmc systran• cmu• ••upc-mr • 0.1 ntt• - nrc• •utd lcc -0.0 -0.1 rali• -0.2 -0.3 -0.4 -0.5 •upv B LEU -0.6 B LEU Spanish-English In Domain ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/koehn-2006-manual-automatic-evaluation-machine-translation/