Elamur Research⌕
☰
home / papers / 2011 / Findings of the 2011 Workshop on Statistical Machine Translation

Findings of the 2011 Workshop on Statistical Machine Translation

Summary

This report documents the outcomes of the WMT11 shared tasks: translation between English and Czech, French, German and Spanish, combining outputs of several MT systems, and evaluation metrics judged against human preferences. The organizers built new news test sets from 110 articles that were professionally translated. They released parallel and monolingual training corpora, development data and two open-source baseline toolkits. They then ran a large manual evaluation covering 148 primary system entries and 41 system combination entries, and used the human rankings to measure how well 21 automatic metrics agree with people. Two tasks were new this year. The first was translating Haitian Creole SMS messages sent to an emergency service after the 2010 earthquake into English, a low-resource and informal setting. The second was a pilot "tunable metrics" task, in which a fixed Urdu-English system was optimized toward different metrics. In that pilot, the system tuned on BLEU was judged about as good as systems tuned on newer metrics. All data, outputs and human judgments are public, which makes the workshop a shared benchmark resource.

Key points

  • The evaluation covered 148 primary entries, including 46 outputs crawled from online, commercial and rule-based systems, plus 60 contrastive entries and 41 system combination entries. Correlation with human judgments was measured for 21 metrics.
  • Test data came from 110 news articles from December 2010, drawn from Czech, English, French, German and Spanish outlets. The agency CEET translated them directly, without a pivot language.
  • Training resources included Europarl, News Commentary, the United Nations corpus, the 10⁹ French-English corpus and CzEng. Moses-style phrase-based and parsing-based baseline toolkits were also provided.
  • For system combination, the data was split into a tuning set and a held-out test set:
    • European pairs: 1,003 tuning segments and 2,000 test segments.
    • Haitian Creole: 674 tuning segments and 600 test segments.
  • In the tunable metrics pilot (Urdu-English):
    • The BLEU-tuned system scored 0.79 on the ≥others measure, roughly level with CMU-METEOR, RWTH-CDER and others.
    • SHEFFIELD-ROSE produced overly long output and scored only 0.05.
    • Ties were frequent: 38% of comparisons overall and 47% without the reference, against 21% for European pairs.

Interesting ideas

  • I1. Fix the system, vary only the tuning metric. This isolates the metric's effect on output quality. It is a clean way to test whether a metric is useful in practice, beyond just correlating with human rankings.
  • I2. Treat crawled third-party outputs as unconstrained. Online and commercial systems may have seen the test data, so they are labelled and anonymized separately. Any benchmark that mixes open and black-box systems needs this kind of contamination caution.
  • I3. Check rankings under both win-only and win-or-tie scoring. Rankings shifted noticeably between >others and ≥others, so a high tie rate is itself a signal. It suggests annotators may be unable to separate low-quality outputs, and conclusions should be weakened accordingly.
  • I4. Split a tuning portion off the test set for downstream combiners. Releasing that portion with references gives second-stage systems fair tuning data while keeping a truly held-out evaluation part.
  • I5. Build a featured task from real crisis messages. Using actual SMS text from a disaster gives a realistic testbed for low-resource, noisy language. The same approach could suit under-resourced languages or domains in other projects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

135 papers of our library cite it (of the 140673 whose bibliographies we read); 22 of them have a page here, 113 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{callisonburch2011findings,
  title = {{Findings of the 2011 Workshop on Statistical Machine Translation}},
  author = {Callison-Burch, Chris and Koehn, Philipp and Monz, Christof and Zaidan, Omar},
  booktitle = {Proceedings of the Sixth Workshop on Statistical Machine Translation},
  year = {2011},
  url = {https://aclanthology.org/W11-2103/}
}

References

80 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vera Aleksic and Gregor Thurmair. 2011. Personal Translator at WMT2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  2. Alexandre Allauzen, Hélène Bonneau-Maynard, Hai-Son Le, Aurélien Max, Guillaume Wisniewski, François Yvon, Gilles Adda, Josep Maria Crego, Adrien Lardilleux, Thomas Lavergne, and Artem Sokolov. 2011. LIMSI @ WMT11. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  3. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater v.2.0. Journal of Technology, Learning, and Assessment, 4(3):159–174. find
  4. Eleftherios Avramidis, Maja Popović, David Vilar, and Aljoscha Burchardt. 2011. Evaluate with confidence http://statmt.org/wmt11/results.html estimation: Machine ranking of translation outputs using grammatical features. In Proceedings of the Sixth Workshop on Statistical Machine Translation. ↗
  5. Wilker Aziz, Miguel Rios, and Lucia Specia. 2011. Shallow semantic trees for SMT. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  6. Kathryn Baker, Michael Bloodgood, Chris Callison-Burch, Bonnie Dorr, Scott Miller, Christine Piatko, Nathaniel W. Filardo, and Lori Levin. 2010. Semantically-informed syntactic machine translation: A tree-grafting approach. In Proceedings of AMTA. find
  7. Loı̈c Barrault. 2011. MANY improvements for WMT’11. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  8. Ergun Bicici and Deniz Yuret. 2011. RegMT system for machine translation, system combination, and evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  9. Ondřej Bojar and Aleš Tamchyna. 2011. Improving translation model by monolingual data. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  10. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation (WMT07), Prague, Czech Republic. find our card
  11. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation (WMT08), Colmbus, Ohio. find
  12. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009. Findings of the 2009 workshop on statistical machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT09), Athens, Greece. find our card
  13. Chris Callison-Burch, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar F. Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT10), Up… find our card
  14. Boxing Chen and Roland Kuhn. 2011. Amber: A modified bleu, enhanced ranking metric. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  15. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurment, 20(1):37–46. find
  16. Antonio M. Corbı́-Bellot, Mikel L. Forcada, Sergio Ortiz-Rojas, Juan Antonio Pérez-Ortiz, Gema Ramı́rez-Sánchez, Felipe Sánchez-Martı́nez, Iñaki Alegria, Aingeru Mayor, and Kepa Sarasola. 2005. An open-source shallow-transfer machine translation engine for the romance languages of Spain. In Proc… find
  17. Marta R. Costa-jussà and Rafael E. Banchs. 2011. The BM-I2R Haitian-Créole-to-English translation system description for the WMT 2011 evaluation campaign. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  18. Daniel Dahlmeier, Chang Liu, and Hwee Tou Ng. 2011. TESLA at WMT 2011: Translation evaluation and tunable metric. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  19. Michael Denkowski and Alon Lavie. 2011a. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find our card
  20. Michael Denkowski and Alon Lavie. 2011b. METEOR-Tuned Phrase-Based SMT: CMU French-English and Haitian-English Systems for WMT 2011. Technical Report CMU-LTI-11-011, Language Technologies Institute, Carnegie Mellon University. find
  21. Chris Dyer, Kevin Gimpel, Jonathan H. Clark, and Noah A. Smith. 2011. The CMU-ARK German-English translation system. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  22. Vladimir Eidelman, Kristy Hollingshead, and Philip Resnik. 2011. Noisy SMS machine translation in low-density languages. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  23. Christian Federmann and Sabine Hunsicker. 2011. Stochastic parse tree selection for an existing RBMT system. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  24. Robert Frederking, Alexander Rudnicky, and Christopher Hogan. 1997. Interactive speech translation in the DIPLOMAT project. In Proceedings of the ACL-1997 Workshop on Spoken Language Translation. find
  25. Markus Freitag, Gregor Leusch, Joern Wuebker, Stephan Peitz, Hermann Ney, Teresa Herrmann, Jan Niehues, Alex Waibel, Alexandre Allauzen, Gilles Adda, Josep Maria Crego, Bianka Buschbeck, Tonio Wandmacher, and Jean Senellart. 2011. Joint WMT submission of the QUAERO project. In Proceedings of the Six… find
  26. Yoko Futagi, Paul Deane, Martin Chodorow, and Joel Tetreault. 2008. A computational approach to detecting collocation errors in the writing of non-native speakers of English. Computer Assisted Language Learning Journal. find
  27. Jesús González-Rubio and Francisco Casacuberta. 2011. The UPV-PRHLT combination system for WMT 2011. find
  28. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  29. Greg Hanneman and Alon Lavie. 2011. CMU syntax-based machine translation at WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  30. Christian Hardmeier, Jörg Tiedemann, Markus Saers, Marcello Federico, and Mathur Prashant. 2011. The Uppsala-FBK systems at WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  31. Kenneth Heafield and Alon Lavie. 2011. CMU system combination in WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  32. Teresa Herrmann, Mohammed Mediani, Jan Niehues, and Alex Waibel. 2011. The Karlsruhe Institute of Technology translation systems for the WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  33. Sanjika Hewavitharana, Nguyen Bach, Qin Gao, Vamshi Ambati, and Stephan Vogel. 2011. CMU Haitian Creole-English translation system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  34. Maria Holmqvist, Sara Stymne, and Lars Ahrenberg. 2011. Experiments with word alignment, normalization and clause reordering for SMT between English and German. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  35. Chang Hu, Philip Resnik, Yakov Kronrod, Vladimir Eidelman, Olivia Buzek, and Benjamin B. Bederson. 2011. The value of monolingual crowdsourcing in a real-world translation scenario: Simulation using Haitian Creole emergency SMS messages. In Proceedings of the Sixth Workshop on Statistical Machine Tr… find
  36. Matthias Huck, Joern Wuebker, Christoph Schmidt, Markus Freitag, Stephan Peitz, Daniel Stein, Arnaud Dagnelies, Saab Mansour, Gregor Leusch, and Hermann Ney. 2011. The RWTH Aachen machine translation system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  37. Maxim Khalilov and Khalil Sima’an. 2011. ILLC-UvA translation system for EMNLP-WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  38. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings of NAACL 2006 Workshop on Statistical Machine Translation, New York, New York. find our card
  39. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc… find
  40. Oliver Lacey-Hall. 2011. The guardian’s poverty matters blog: How remote teams can help the rapid response to disasters, March. find
  41. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159–174. find
  42. Gregor Leusch and Hermann Ney. 2009. Edit distances with block movements and error rate confidence estimates. Machine Translation, 23:129–140. find
  43. Gregor Leusch, Markus Freitag, and Hermann Ney. 2011. The RWTH system combination system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  44. William Lewis, Robert Munro, and Stephan Vogel. 2011. Crisis MT: Developing a cookbook for MT in crisis situations. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  45. William D. Lewis. 2010. Haitian Creole: How to build and ship an MT engine from scratch in 4 days, 17hours, & 30 minutes. In Proceedings of EAMT 2010. find
  46. Zhifei Li, Chris Callison-Burch, Chris Dyer, Juri Ganitkevitch, Ann Irvine, Sanjeev Khudanpur, Lane Schwartz, Wren Thornton, Ziyuan Wang, Jonathan Weese, and Omar Zaidan. 2010. Joshua 2.0: A toolkit for parsing-based machine translation with syntax, semirings, discriminative training and other goodi… find
  47. Ding Liu and Daniel Gildea. 2005. Syntactic features for evaluation of machine translation. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 25–32. find our card
  48. Chang Liu, Daniel Dahlmeier, and Hwee Tou Ng. 2011. Better evaluation metrics lead to better machine translation. In Proceedings of EMNLP. find
  49. Verónica López-Ludeña and Rubén San-Segundo. 2011. UPM system for the translation task. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  50. Matouš Macháček and Ondřej Bojar. 2011. Approximating a deep-syntactic metric for MT evaluation and tuning. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  51. David Mareček, Rudolf Rosa, Petra Galuščáková, and Ondřej Bojar. 2011. Two-step translation with grammatical post-processing. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  52. Robert Munro. 2010. Crowdsourced translation for emergency response in Haiti: the global collaboration of local knowledge. In Proceedings of the AMTA Workshop on Collaborative Crowdsourcing for Translation. find
  53. Douglas W. Oard and Franz Josef Och. 2003. Rapid-response machine translation for unexpected languages. In Proceedings of MT Summit IX. find
  54. Douglas W. Oard. 2003. The surprise language exercises. ACM Transactions on Asian Language Information Processing, 2(2):79–84. find
  55. Franz Josef Och. 2003. Minimum error rate training for statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL-2003), Sapporo, Japan. find
  56. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL-2002), Philadelphia, Pennsylvania. find our card
  57. Kristen Parton, Joel Tetreault, Nitin Madnani, and Martin Chodorow. 2011. E-rating machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  58. Martin Popel, David Mareček, Nathan Green, and Zdenêk Ẑabokrtský. 2011. Influence of parser choice on dependency-based MT. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  59. Maja Popović, David Vilar, Eleftherios Avramidis, and Aljoscha Burchardt. 2011. Evaluation without references: IBM1 scores as evaluation metrics. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  60. Maja Popović. 2011. Morphemes and POS tags for ngram based evaluation metrics. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  61. Marion Potet, Raphaël Rubino, Benjamin Lecouteux, Stéphane Huet, Laurent Besacier, Hervé Blanchon, and Fabrice Lefèvre. 2011. The LIGA (LIG/LIA) machine translation system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  62. Mark Przybocki, Kay Peterson, and Sebastian Bronsart. 2008. Official results of the NIST 2008 “Metrics for MAchine TRanslation” challenge (Metrics-MATR08). In AMTA-2008 workshop on Metrics for Machine Translation, Honolulu, Hawaii. find
  63. Miguel Rios, Wilker Aziz, and Lucia Specia. 2011. TINE: A metric to assess MT adequacy. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  64. Christian Rishøj and Anders Søgaard. 2011. Factored translation with unsupervised word clusters. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  65. Antti-Veikko Rosti, Bing Zhang, Spyros Matsoukas, and Richard Schwartz. 2011. Expected BLEU training for graphs: BBN system description for WMT11 system combination task. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  66. Vı́ctor M. Sánchez-Cartagena, Felipe Sánchez-Martı́nez, and Juan Antonio Pérez-Ortiz. 2011. The Universitat d’Alacant hybrid machine translation system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  67. Holger Schwenk, Patrik Lambert, Loı̈c Barrault, Christophe Servan, Sadaf Abdul-Rauf, Haithem Afli, and Kashif Shah. 2011. LIUM’s SMT machine translation systems for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  68. Rico Sennrich. 2011. The UZH system combination system for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  69. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Biennial Conference of the Association for Machine Translation in the Americas (AMTA-2006), Cambridge, Massachusetts. find our card
  70. Xingyi Song and Trevor Cohn. 2011. Regression and ranking based optimisation for sentence level MT evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  71. Lucia Specia, Dhwaj Raj, and Marco Turchi. 2010. Machine translation evaluation versus quality estimation. Machine Translation, 24(1):39–50. find
  72. Sara Stymne. 2011. Spell checking techniques for replacement of unknown words and data cleaning for Haitian Creole SMS translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  73. Joel Tetreault and Martin Chodorow. 2008. The ups and downs of preposition error detection. In Proceedings of COLING, Manchester, UK. find
  74. Jonathan Weese, Juri Ganitkevitch, Chris Callison-Burch, Matt Post, and Adam Lopez. 2011. Joshua 3.0: Syntax-based machine translation with the Thrax grammar extractor. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  75. Eric Wehrli, Luka Nerima, and Yves Scherrer. 2009. Deep linguistic multilingual translation and bilingual dictionaries. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 90–94. find
  76. Daguang Xu, Yuan Cao, and Damianos Karakos. 2011a. Description of the JHU system combination scheme for WMT 2011. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  77. Jia Xu, Hans Uszkoreit, Casey Kennington, David Vilar, and Xiaojun Zhang. 2011b. DFKI hybrid machine translation system for WMT 2011 - on the integration of SMT and RBMT. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  78. Omar F. Zaidan. 2009. Z-MERT: A fully configurable open source tool for minimum error rate training of machine translation systems. The Prague Bulletin of Mathematical Linguistics, 91:79–88. find
  79. Francisco Zamora-Martinez and Maria Jose Castro-Bleda. 2011. CEU-UPV English-Spanish system for WMT11. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  80. Daniel Zeman. 2011. Hierarchical phrase-based MT at the Charles University for the WMT 2011 shared task. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/burch-2011-findings-2011-workshop-statistical-machine/