Elamur Research
☰
home / papers / 2012 / Findings of the 2012 Workshop on Statistical Machine Translation

Findings of the 2012 Workshop on Statistical Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W12-3102.pdf

Summary

This report covers the shared tasks of WMT12, the seventh Workshop on Statistical Machine Translation, held at NAACL 2012. It has three goals: to benchmark current MT systems, to check how well automatic metrics match human judgments, and to launch a new challenge. In the main task, systems translated news text between English and Czech, French, German and Spanish. The organizers supplied training corpora and freshly translated test sets, and also collected outputs from anonymized commercial, online and rule-based systems. Human ranking of outputs was the official measure of quality, and it was used to validate 12 automatic metrics. Quality estimation replaced the earlier system combination task. Its participants predicted how good Moses English-Spanish output was without seeing reference translations, either by ranking sentences or by scoring them. The main results were the human-based system rankings, metric correlation figures, and a new state of the art for quality estimation. All data, outputs and human judgments are publicly released, which makes the collection a lasting resource for MT and evaluation research.

Key points

  • Scale: 103 systems from 34 teams were evaluated by hand. Annotators, including Amazon Mechanical Turk workers screened with control items, produced 61,695 ranking labels and about 336 hours of usable work.
  • Test data: 99 news articles from November 2011 were translated directly by the agency CEET. The authors noted some translation errors and recommend a manual review in future years.
  • Training resources: These included Europarl, News Commentary, United Nations, the 10⁹ Word Parallel Corpus (French-English) and CzEng, plus large monolingual news corpora for language models.
  • Quality estimation task: 11 teams took part. M5P regression trees and SVR worked best. M5P produced compact models that resisted overfitting, while SVR needed feature selection and tuning to avoid it.
  • Ranking metric: The new DeltaAvg metric correlated well with Spearman's rank correlation and proved suitable for the ranking subtask. Scoring turned out harder than ranking, because even professional annotators struggle to give consistent absolute scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

203 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 184 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{callisonburch2012findings,
  title = {{Findings of the 2012 Workshop on Statistical Machine Translation}},
  author = {Callison-Burch, Chris and Koehn, Philipp and Monz, Christof and Post, Matt and Soricut, Radu and Specia, Lucia},
  booktitle = {Proceedings of the Seventh Workshop on Statistical Machine Translation},
  year = {2012},
  url = {https://aclanthology.org/W12-3102/}
}

References

61 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eleftherios Avramidis. 2012. Quality estimation for machine translation output using linguistic analysis and decoding features. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  2. Ondřej Bojar, Miloš Ercegovčević, Martin Popel, and Omar Zaidan. 2011. A grain of salt for the wmt manual evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 1–11, Edinburgh, Scotland, July. Association for Computational Linguistics. find
  3. Ondrej Bojar, Bushra Jawaid, and Amir Kamran. 2012. Probes in a taxonomy of factored phrase-based models. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  4. Christian Buck. 2012. Black box features for the WMT 2012 quality estimation shared task. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  5. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation (WMT07), Prague, Czech Republic. find
  6. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation (WMT08), Colmbus, Ohio. find
  7. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009. Findings of the 2009 workshop on statistical machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT09), Athens, Greece. find
  8. Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar F. Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT10), Up… find
  9. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22–64, Edinburgh, Scotland, July. find
  10. Chris Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using Amazon’s Mechanical Turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP-2009), Singapore. find
  11. Julio Castillo and Paula Estrella. 2012. Semantic textual similarity for MT evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  12. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27. Software available at http://www. csie.ntu.edu.tw/˜cjlin/libsvm. find
  13. Boxing Chen, Roland Kuhn, and George Foster. 2012. Improving amber, an MT evaluation metric. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  14. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurment, 20(1):37–46. find
  15. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  16. Michael Denkowski, Greg Hanneman, and Alon Lavie. 2012. The CMU-avenue French-English translation system. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  17. Ondřej Dušek, Zdeněk Žabokrtský, Martin Popel, Martin Majliš, Michal Novák, and David Mareček. 2012. Formemes in English-Czech deep syntactic mt. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find our card
  18. Mariano Felice and Lucia Specia. 2012. Linguistic features for quality estimation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  19. Mark Fishel, Rico Sennrich, Maja Popović, and Ondřej Bojar. 2012. TerrorCat: a translation error categorization-based MT quality metric. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  20. Lluis Formiga, Carlos A. Henrı́quez Q., Adolfo Hernández, José B. Mariño, Enric Monte, and José A. R. Fonollosa. 2012. The TALP-UPC phrase-based translation systems for WMT12: Morphology simplification and domain adaptation. In Proceedings of the Seventh Workshop on Statistical Machine Translati… find
  21. Juri Ganitkevitch, Yuan Cao, Jonathan Weese, Matt Post, and Chris Callison-Burch. 2012. Joshua 4.0: Packing, PRO, and paraphrases. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  22. Ulrich Germann. 2012. Syntax-aware phrase-based statistical machine translation: System description. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  23. Jesús González-Rubio, Alberto Sanchı́s, and Francisco Casacuberta. 2012. PRHLT submission to the WMT12 quality estimation task. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  24. Francisco Guzman, Preslav Nakov, Ahmed Thabet, and Stephan Vogel. 2012. QCRI at WMT12: Experiments in Spanish-English and German-English machine translation of news text. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational… find
  25. Christian Hardmeier, Joakim Nivre, and Jörg Tiedemann. 2012. Tree kernels for machine translation quality estimation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  26. Christian Hardmeier. 2011. Improving machine translation quality prediction with syntactic tree kernels. In Proceedings of the 15th conference of the European Association for Machine Translation, pages 233–240, Leuven, Belgium. find
  27. Matthias Huck, Stephan Peitz, Markus Freitag, Malte Nuhn, and Hermann Ney. 2012. The RWTH aachen machine translation system for WMT 2012. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  28. Sabine Hunsicker, Chen Yu, and Christian Federmann. 2012. Machine learning for hybrid machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  29. Philipp Koehn and Barry Haddow. 2012. Towards effective use of training data in statistical machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  30. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings of NAACL 2006 Workshop on Statistical Machine Translation, New York, New York. find
  31. Philipp Koehn. 2004. Statistical Significance Tests for Machine Translation Evaluation. In Proceedings of the Empirical Methods in Natural Language Processing Conference. find
  32. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159–174. find
  33. David Langlois, Sylvain Raybaud, and Kamel Smaı̈li. 2012. LORIA system for the WMT12 quality estimation shared task. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  34. Hai-Son Le, Thomas Lavergne, Alexandre Allauzen, Marianna Apidianaki, Li Gong, Aurélien Max, Artem Sokolov, Guillaume Wisniewski, and François Yvon. 2012. LIMSI @ WMT12. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational… find
  35. Verónica López-Ludeña, Rubén San-Segundo, and Juan M. Montero. 2012. UPM system for WMT 2012. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  36. Adam Lopez. 2012. Putting human assessments of machine translation systems in order. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  37. Matouš Macháček and Ondej Bojar. 2011. Approximating a deep-syntactic metric for mt evaluation and tuning. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 373–379, Edinburgh, Scotland, July. Association for Computational Linguistics. find
  38. Freitag Markus, Peitz Stephan, Huck Matthias, Ney Hermann, Niehues Jan, Herrmann Teresa, Waibel Alex, Hai-son Le, Lavergne Thomas, Allauzen Alexandre, Buschbeck Bianka, Crego Joseph Maria, and Senellart Jean. 2012. find
  39. Joint WMT 2012 submission of the QUAERO project. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  40. Alexander Molchanov. 2012. PROMT deephybrid system for WMT12 shared translation task. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  41. Erwan Moreau and Carl Vogel. 2012. Quality estimation: an experimental study using unsupervised similarity measures. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  42. Jan Niehues, Yuqi Zhang, Mohammed Mediani, Teresa Herrmann, Eunah Cho, and Alex Waibel. 2012. The karlsruhe institute of technology translation systems for the WMT 2012. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational L… find
  43. Daniele Pighin, Meritxell González, and Lluı́s Màrquez. 2012. The upc submission to the WMT 2012 shared task on quality estimation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  44. Maja Popovic. 2012. Class error rates for evaluation of machine translation output. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  45. Sylvain Raybaud, David Langlois, and Kamel Smaı̈li. 2011. ”This sentence is wrong.” Detecting errors in machine-translated sentences. Machine Translation, 25(1):1–34. find
  46. Majid Razmara, Baskaran Sankaran, Ann Clifton, and Anoop Sarkar. 2012. Kriya - the SFU system for translation task at WMT-12. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  47. Rudolf Rosa, David Mareček, and Ondřej Dušek. 2012. DEPFIX: A system for automatic correction of czech MT outputs. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  48. Raphael Rubino, Jennifer Foster, Joachim Wagner, Johann Roturier, Rasul Samad Zadeh Kaljahi, and Fred Hollowood. 2012. DCU-Symantec submission for the WMT 2012 quality estimation task. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for… find
  49. Christophe Servan, Patrik Lambert, Anthony Rousseau, Holger Schwenk, and Loı̈c Barrault. 2012. LIUM’s smt machine translation systems for WMT 2012. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  50. Xingyi Song and Trevor Cohn. 2011. Regression and ranking based optimisation for sentence level MT evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  51. Radu Soricut, Nguyen Bach, and Ziyuan Wang. 2012. The SDL Language Weaver systems in the WMT12 quality estimation shared task. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  52. Lucia Specia, Marco Turchi, Nicola Cancedda, Marc Dymetman, and Nello Cristianini. 2009. Estimating the Sentence-Level Quality of Machine Translation Systems. In Proceedings of the 13th Conference of the European Association for Machine Translation, pages 28–37, Barcelona. find
  53. Lucia Specia. 2011. Exploiting Objective Annotations for Measuring Translation Post-editing Effort. In Proceedings of the 15th Conference of the European Association for Machine Translation, pages 73–80, Leuven. find
  54. Aleš Tamchyna, Petra Galuščáková, Amir Kamran, Miloš Stanojević, and Ondřej Bojar. 2012. Selecting data for English-to-Czech machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  55. David Vilar. 2012. DFKI’s smt system for WMT 2012. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  56. Mengqiu Wang and Christopher Manning. 2012. SPEDE: Probabilistic edit distance metrics for MT evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  57. Eric Wehrli, Luka Nerima, and Yves Scherrer. 2009. Deep linguistic multilingual translation and bilingual dictionaries. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 90–94. find
  58. Philip Williams and Philipp Koehn. 2012. GHKM rule extraction and scope-3 parsing in Moses. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  59. Chunyang Wu and Hai Zhao. 2012. Regression with phrase indicators for estimating MT quality. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  60. Daniel Zeman. 2012. Data issues of the multilingual translation matrix. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Montreal, Canada, June. Association for Computational Linguistics. find
  61. CU - BOJAR CU - BOJAR – .59? .44 .36? .36? .56? 0.53 JHU ONLINE - A ONLINE - B UEDIN UK \> others Table 14: Head to head comparison for Czech-English systems find