Elamur Research⌕
☰
home / papers / 2009 / Findings of the 2009 Workshop on Statistical Machine Translation

Findings of the 2009 Workshop on Statistical Machine Translation

Summary

WMT09 was the fourth edition of a shared evaluation campaign for machine translation. It aimed to measure the state of the art, release common data with published results, and improve how translation quality is assessed. It ran three tasks: translation between English and German, Spanish, French, Czech and Hungarian; combining outputs from several MT systems; and predicting human quality judgments with automatic metrics. The organizers simplified the setup. They dropped the Europarl in-domain test set, used news text only, and made sentence-level ranking the main human evaluation. Training data grew substantially, mainly through a crawled French–English corpus of roughly a billion words. Humans evaluated 87 MT systems and 22 combination entries, and their rankings were used to test more than 20 automatic metrics. The workshop also introduced a new protocol in which annotators edit system output and others then judge whether the edits are correct. ULC correlated best with humans for into-English translation, while sentence-level prediction remained close to chance. All data, outputs and judgments were made public.

Key points

  • Participation: 22 groups from 20 institutions submitted 87 primary and 42 secondary systems. The organizers also evaluated 7 commercial rule-based systems and Google's online system. The baseline was Moses, which 12 groups built on.
  • Test set: 136 news articles from late 2008 were translated directly into seven languages, including Italian, without pivoting through another language. This came to about 80,000 words in 3,027 sentences, at a cost of roughly 31,700 EUR (just over $0.08 per word).
  • French–English 10⁹ corpus:
    • A crawl of more than 1TB of bilingual sites produced about 40M files.
    • Documents were paired by rewriting URLs (e.g., swapping fr for en).
    • Sentences were aligned with Moore's aligner, which combines sentence length and IBM Model 1, giving 177M pairs.
    • Cleaning reduced this to 105M pairs.
    • Bloom-filter de-duplication left about 28M pairs.
  • Other training resources: Europarl, News Commentary, Hunglish, CzEng, and large monolingual news corpora for language modelling.
  • Metric results: ULC was best for into-English (MaxSim and RTE close behind), and TERp and wpBleu were best for out-of-English.
  • Annotator agreement:
    • Ranking agreement (Kappa) was only fair.
    • Binary correctness judgments of edits reached moderate agreement.
    • Agreement of annotators with themselves ranged from moderate to substantial.

Interesting ideas

  • I1. Edit-then-verify evaluation: Annotators fix MT output without seeing the source or reference, and others then check whether the fixed version is correct. This measures how understandable a translation is, and it could be applied to any generation task where fluency can hide meaning errors.
  • I2. Cheap parallel-corpus mining by URL rewriting: Language codes in URLs can pair documents across languages. Dropping pairs whose two sides are identical then acts as a crude substitute for language identification. This is useful for building corpora for lower-resource pairs, such as Ukrainian with another language, from government and organization websites.
  • I3. Discarding annotator warm-up: Removing each judge's first judgments (up to 50) raised agreement, and so did excluding the least consistent judges. Both are simple quality filters for any crowdsourced annotation.
  • I4. System-level priors for sentence-level scores: Metrics did much better when each sentence received its system's overall score. This suggests that combining a prior about the system with sentence-level evidence could improve segment-level quality estimation.
  • I5. Shuffling sentences in released monolingual data: Randomizing sentence order in crawled news reduces copyright concerns while keeping the text usable for language modelling. This is a practical way to share web-derived corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

136 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 116 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{callisonburch2009findings,
  title = {{Findings of the 2009 Workshop on Statistical Machine Translation}},
  author = {Callison-Burch, Chris and Koehn, Philipp and Monz, Christof and Schroeder, Josh},
  booktitle = {Proceedings of the Fourth Workshop on Statistical Machine Translation},
  year = {2009},
  url = {https://aclanthology.org/W09-0401/}
}

References

51 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abhaya Agarwal and Alon Lavie. 2008. Meteor, M-BLEU and M-TER: Evaluation metrics for high-correlation with human rankings of machine translation output. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 115–118, Columbus, Ohio, June. Association for Computational Lingui… find
  2. Joshua Albrecht and Rebecca Hwa. 2007a. A re-examination of machine learning approaches for 8 http://www.statmt.org/wmt09/results. html sentence-level MT evaluation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL-2007), Prague, Czech Republic. ↗
  3. Joshua Albrecht and Rebecca Hwa. 2007b. Regression for sentence-level MT evaluation with pseudo references. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL-2007), Prague, Czech Republic. find
  4. Alexandre Allauzen, Josep Crego, Aurélien Max, and Fran cois Yvon. 2009. LIMSI’s statistical translation systems for WMT’09. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  5. Ondřej Bojar, David Mareček, Václav Novák, Martin Popel, Jan Ptáček, Jan Rouš, and Zdeněk Žabokrtský. 2009. English-Czech MT in 2008. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  6. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation (WMT07), Prague, Czech Republic. find our card
  7. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation (WMT08), Colmbus, Ohio. find
  8. Marine Carpuat. 2009. Toward using morphology in French-English phrase-based SMT. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  9. Yee Seng Chan and Hwee Tou Ng. 2008. An automatic metric for machine translation evaluation based on maximum similary. In In the Metrics-MATR Workshop of AMTA-2008, Honolulu, Hawaii. find
  10. Yu Chen, Michael Jellinghaus, Andreas Eisele, Yi Zhang, Sabine Hunsicker, Silke Theison, Christian Federmann, and Hans Uszkoreit. 2009. Combining multi-engine translations with moses. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Com… find
  11. Jinhua Du, Yifan He, Sergio Penkale, and Andy Way. 2009. MATREX: The DCU MT system for WMT 2009. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  12. Loı̈c Dugast, Jean Senellart, and Philipp Koehn. 2009. Statistical post editing and dictionary extraction: Systran/Edinburgh submissions for ACL-WMT2009. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  13. Chris Dyer, Hendra Setiawan, Yuval Marton, and Philip Resnik. 2009. The University of Maryland statistical machine translation system for the fourth workshop on machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Compu… find
  14. Jason Eisner and Roy W. Tromble. 2006. Local search with very large-scale neighborhoods for optimal permutations in machine translation. In Proceedings of the Human Language Technology Conference of the North American chapter of the Association for Computational Linguistics (HLT/NAACL-2006), New Yor… find
  15. Christian Federmann, Silke Theison, Andreas Eisele, Hans Uszkoreit, Yu Chen, Michael Jellinghaus, and Sabine Hunsicker. 2009. Translation combination using factored word substitution. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Com… find
  16. Alexander Fraser. 2009. Experiments in morphosyntactic processing for translating to and from German. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  17. Jesús Giménez and Lluı́s Màrquez. 2007. Linguistic features for automatic evaluation of heterogenous MT systems. In Proceedings of ACL Workshop on Machine Translation. find
  18. Jesús Giménez and Lluı́s Màrquez. 2008. A smorgasbord of features for automatic MT evaluation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 195–198. find
  19. Greg Hanneman, Vamshi Ambati, Jonathan H. Clark, Alok Parlikar, and Alon Lavie. 2009. An improved statistical transfer system for French-English machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguis… find
  20. Kenneth Heafield, Greg Hanneman, and Alon Lavie. 2009. Machine translation system combination with flexible word ordering. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  21. Almut Silja Hildebrand and Stephan Vogel. 2009. CMU system combination for WMT’09. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  22. Maria Holmqvist, Sara Stymne, Jody Foo, and Lars Ahrenberg. 2009. Improving alignment for SMT by reordering and augmenting the training corpus. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  23. Philipp Koehn and Barry Haddow. 2009. Edinburgh’s submission to all tracks of the WMT2009 shared task with reordering and speed improvements to Moses. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  24. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings of NAACL 2006 Workshop on Statistical Machine Translation, New York, New York. find our card
  25. Philipp Koehn, Nicola Bertoldi, Ondrej Bojar, Chris Callison-Burch, Alexandra Constantin, Brooke Cowan, Chris Dyer, Marcello Federico, Evan Herbst, Hieu Hoang, Christine Moran, Wade Shen, and Richard Zens. 2007. Open source toolkit for statistical machine translation: Factored translation models and… find
  26. Kamil Kos and Ondřej Bojar. 2009. Evaluation of Machine Translation Metrics for Czech as the Target Language. Prague Bulletin of Mathematical Linguistics, 92. in print. find
  27. Alex Kulesza and Stuart M. Shieber. 2004. A learning approach to improving sentence-level MT evaluation. In Proceedings of the 10th International Conference on Theoretical and Methodological Issues in Machine Translation, Baltimore, MD, October 4–6. find
  28. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159–174. find
  29. Gregor Leusch and Hermann Ney. 2008. BLEUSP, PINVWER, CDER: Three improved MT evaluation measures. In In the Metrics-MATR Workshop of AMTA-2008, Honolulu, Hawaii. find
  30. Gregor Leusch, Evgeny Matusov, and Hermann Ney. 2009. The RWTH system combination system for WMT 2009. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  31. Zhifei Li, Chris Callison-Burch, Chris Dyer, Juri Ganitkevitch, Sanjeev Khudanpur, Lane Schwartz, Wren Thornton, Jonathan Weese, and Omar Zaidan. 2009. Joshua: An open source toolkit for parsing-based machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athe… find our card
  32. Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-2004), Barcelona, Spain. find our card
  33. Robert C. Moore. 2002. Fast and accurate sentence alignment of bilingual corpora. In Proceedings of the 5th Biennial Conference of the Association for Machine Translation in the Americas (AMTA-2002), Tiburon, California. find
  34. Preslav Nakov and Hwee Tou Ng. 2009. NUS at WMT09: Domain adaptation experiments for English-Spanish machine translation of news commentary text. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  35. Jan Niehues, Teresa Herrmann, Muntsin Kolss, and Alex Waibel. 2009. The Universität Karlsruhe translation system for the EACL-WMT 2009. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  36. NIST. 2008. Evaluation plan for gale go/no-go phase 3 / phase 3.5 translation evaluations. June 18, 2008. find
  37. Attila Novák. 2009. Morphologic’s submission for the WMT 2009 shared task. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  38. Sebastian Pado, Michel Galley, Dan Jurafsky, and Christopher D. Manning. 2009. Machine translation evaluation with textual entailment features. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  39. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL-2002), Philadelphia, Pennsylvania. find our card
  40. Michael Paul, Andrew Finch, and Eiichiro Sumita. 2009. NICT@WMT09: Model adaptation and transliteration for Spanish-English SMT. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  41. Maja Popovic and Hermann Ney. 2009. Syntax-oriented evaluation measures for machine translation output. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  42. Maja Popovic, David Vilar, Daniel Stein, Evgeny Matusov, and Hermann Ney. 2009. The RWTH machine translation system for WMT 2009. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  43. Mark Przybocki, Kay Peterson, and Sebastien Bronsart. 2008. Official results of the NIST 2008 “Metrics for MAchine TRanslation” challenge (MetricsMATR08). http://nist.gov/speech/tests/metricsmatr/2008/results/. ↗
  44. José A. R. Fonollosa, Maxim Khalilov, Marta R. Costa-jussá, José B. Mariño, Carlos A. Henráquez Q., Adolfo Hernández H., and Rafael E. Banchs. 2009. The TALP-UPC phrase-based translation system for EACL-WMT 2009. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens,… find
  45. Antti-Veikko Rosti, Bing Zhang, Spyros Matsoukas, and Richard Schwartz. 2009. Incremental hypothesis alignment with flexible matching for building confusion networks: BBN system description for WMT09 system combination task. In Proceedings of the Fourth Workshop on Statistical Machine Translation, A… find
  46. Josh Schroeder, Trevor Cohn, and Philipp Koehn. 2009. Word lattices for multi-source translation. In 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL-2009), Athens, Greece. find
  47. Holger Schwenk, Sadaf Abdul Rauf, Loic Barrault, and Jean Senellart. 2009. SMT and SPE machine translation systems for WMT’09. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find
  48. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Biennial Conference of the Association for Machine Translation in the Americas (AMTA-2006), Cambridge, Massachusetts. find our card
  49. Matthew Snover, Nitin Madnani, Bonnie Dorr, and Richard Schwartz. 2009. Fluency, adequacy, or HTER? exploring different human judgments with a tunable MT metric. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics… find our card
  50. David Talbot and Miles Osborne. 2007. Smoothed Bloom filter language models: Tera-scale lms on the cheap. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), Prague, Czech Republic. find
  51. Eric Wehrli, Luka Nerima, and Yves Scherrer. 2009. Deep linguistic multilingual translation and bilingual dictionaries. In Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/burch-2009-findings-2009-workshop-statistical-machine/