Elamur Research
☰
home / papers / 2016 / Findings of the 2016 Conference on Machine Translation

Findings of the 2016 Conference on Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W16-2301.pdf

Summary

This overview reports the results of the WMT16 shared tasks, the first edition held as a full Conference on Machine Translation. The campaign benchmarks current MT and related technologies on common data. It covered five translation tasks and three evaluation tasks, plus automatic post-editing (APE) and bilingual document alignment. This paper details news, IT-domain and biomedical translation, quality estimation and APE.

The news task paired English with Czech, German, Finnish, Russian, Romanian and Turkish in both directions. Romanian and Turkish were added as lower-resource pairs. Each test set had about 3,000 sentences of professionally translated news. Systems were scored with automatic metrics and by researcher judges. The official ranking came from relative ranking processed with TrueSkill. A direct assessment on a 100-point scale was piloted alongside it.

In APE, most entries clearly beat a baseline that went unbeaten the previous year, and neural approaches entered the task. Human post-edits were still preferred over all systems. All data, outputs and judgments were released publicly, which makes the campaign a reusable resource for MT and evaluation research.

Key points

  • News task: 102 systems from 24 institutions were submitted, plus 36 anonymized online systems (ONLINE-A, B, F, G) from 4 services. New training resources included Romanian Europarl, SETIMES2 (OPUS), CzEng v1.6pre and News Commentary v11.
  • Human evaluation: 150 annotators from 33 research groups completed 10,833 HITs. These yielded 569,287 pairwise comparisons, 75.2% more than the expected 324,990.
  • New tasks: The IT-domain task received 31 submissions across 7 directions, each going out of English, with Basque, Bulgarian, Czech, Dutch, German, Portuguese and Spanish as the target languages. The biomedical task used Scielo titles and abstracts for Portuguese, Spanish and French paired with English, and drew 15 submissions from 5 institutions.
  • Quality estimation and APE: Quality estimation had 3 subtasks (sentence, word/phrase and document level) with 14 teams and 39 entries. APE covered English→German IT data with 6 teams and 11 entries.
  • APE results: The best APE system improved TER by up to −3.24 and BLEU by up to +5.54. Annotator agreement measured by Fleiss' κ was 0.481. System outputs can be compared interactively with MT-ComparEval at wmt.ufal.cz.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

398 papers of our library cite it (of the 140673 whose bibliographies we read); 38 of them have a page here, 360 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-2301

@inproceedings{bojar2016findings,
  title = {{Findings of the 2016 Conference on Machine Translation}},
  author = {Bojar, Ondřej and Chatterjee, Rajen and Federmann, Christian and Graham, Yvette and Haddow, Barry and Huck, Matthias and Yepes, Antonio Jimeno and Koehn, Philipp and Logacheva, Varvara and Monz, Christof and Negri, Matteo and Névéol, Aurélie and Neves, Mariana and Popel, Martin and Post, Matt and Rubino, Raphael and Scarton, Carolina and Specia, Lucia and Turchi, Marco and Verspoor, Karin and Zampieri, Marcos},
  booktitle = {Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers},
  year = {2016},
  doi = {10.18653/v1/w16-2301},
  url = {https://doi.org/10.18653/v1/w16-2301}
}

References

121 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Béchara, H., Ma, Y., and van Genabith, J. (2011). Statistical Post-Editing for a Statistical MT System. In Proceedings of the 13th Machine Translation Summit, pages 308–315, Xiamen, China. find
  2. Abdelsalam, A., Bojar, O., and El-Beltagy, S. (2016). find
  3. Bilingual Embeddings and Word Align-Beck, D., Vlachos, A., Paetzold, G., and Specia, L. (2016). SHEF-MIME: Word-level Quality on Statistical Machine Translation, pages 12– 58, Baltimore, Maryland, USA. Association for Computational Linguistics. find
  4. Estimation Using Imitation Learning. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  5. Bojar, O., Chatterjee, R., Federmann, C., Haddow, B., Huck, M., Hokamp, C., Koehn, P., Logacheva, V., Monz, C., Negri, M., Post, M., Scarton, C., Specia, L., and Turchi, M. (2015). Findings of the 2015 workshop on statistical machine translation. In Proceedings of the Tenth Workshop on Statistical M… find our card
  6. Bektaş, E., Yilmaz, E., Mermer, C., and Durgar El-Kahlout, . (2016). TÜBTAK SMT System Submission for WMT2016. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  7. Berard, A., Servan, C., Pietquin, O., and Besacier, L. (2016). MultiVec: a Multilingual and Multi-level Representation Learning Toolkit for NLP. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). find
  8. Bojar, O., Dušek, O., Kocmi, T., Libovický, J., Novák, M., Popel, M., Sudarikov, R., and Variš, D. (2016a). CzEng 1.6: Enlarged Czech-English Parallel Corpus with Processing Tools Dockered. In Text, Speech and Dialogue: 19th International Conference, TSD 2016, Brno, Czech Republic, September 12-… find
  9. Biçici, E. and Way, A. (2015). Referential translation machines for predicting semantic similarity. Language Resources and Evaluation, pages 1–27. find
  10. Bojar, O., Ercegovčević, M., Popel, M., and Zaidan, O. (2011). A grain of salt for the wmt manual evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 1–11, Edinburgh, Scotland. Association for Computational Linguistics. find
  11. Bicici, E. (2016a). ParFDA for Instance Selection for Statistical Machine Translation. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  12. Bicici, E. (2016b). Referential Translation Machines for Predicting Translation Performance. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  13. Bojar, O., Graham, Y., , and Stanojević, A. K. M. (2016b). Results of the WMT16 Metrics Shared Task . In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find our card
  14. Blain, F., Song, X., and Specia, L. (2016). Sheffield Systems for the English-Romanian WMT Translation Task. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  15. Bradbury, J. and Socher, R. (2016). MetaMind Neural Machine Translation System for WMT 2016. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  16. Buck, C., Heafield, K., and Van Ooyen, B. (2014). N-gram counts and language models from the common crawl. LREC, 2:4. find
  17. Bojar, O., Buck, C., Callison-Burch, C., Federmann, C., Haddow, B., Koehn, P., Monz, C., Post, M., Soricut, R., and Specia, L. (2013). Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 1–44, Sofia, Bulgar… find our card
  18. Buck, C. and Koehn, P. (2016). Findings of the WMT 2016 Bilingual Document Alignment Shared Task. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  19. Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., Soricut, R., Specia, L., and Tamchyna, A. (2014). Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop C. de Souza, J. G., Buck,… find our card
  20. C. de Souza, J. G., González-Rubio, J., Buck, C., Turchi, M., and Negri, M. (2014). FBK-UPV-UEdin participation in the WMT14 Quality Estimation shared-task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, MD, USA. find
  21. Chatterjee, R., C. de Souza, J. G., Negri, M., and Turchi, M. (2016). The FBK Participation in the WMT 2016 Automatic Post-editing Shared Task. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  22. Callison-Burch, C., Fordyce, C. S., Koehn, P., Monz, C., and Schroeder, J. (2007). (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 136–158, Prague, Czech Republic. Association for Computational Linguistics. find
  23. Chatterjee, R., Turchi, T., and Negri, M. (2015a). The FBK Participation in the WMT15 Automatic Post-editing Shared Task. In Proceedings of the 10th Workshop on Statistical Machine Translation (WMT). find
  24. Chatterjee, R., Weller, M., Negri, M., and Turchi, M. (2015b). Exploring the Planet of the APEs: a Comparative Study of State-of-the-art Methods for MT Automatic Post-Editing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics), Beijing, China. find
  25. Callison-Burch, C., Fordyce, C. S., Koehn, P., Monz, C., and Schroeder, J. (2008). Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 70–106, Columbus, Ohio. Association for Computational Linguistics. find
  26. Chung, J., Cho, K., and Bengio, Y. (2016). NYU-MILA Neural Machine Translation Systems for WMT16. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  27. Callison-Burch, C., Koehn, P., Monz, C., Peterson, K., Przybocki, M., and Zaidan, O. (2010). Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, page… find
  28. Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37–46. find
  29. Costa-jussà, M. R., España Bonet, C., Madhyastha, P., Escolano, C., and Fonollosa, J. A. R. (2016). The TALP–UPC Spanish–English WMT Biomedical Task: Bilingual Embeddings and Char-based Neural Language Model Rescoring in a Phrase-based System. In Proceedings of the First Conference on Machine Tran… find
  30. Callison-Burch, C., Koehn, P., Monz, C., Post, M., Soricut, R., and Specia, L. (2012). Findings of the 2012 workshop on statistical machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10– 48, Montreal, Canada. Association for Computational Linguistic… find our card
  31. Cuong, H., Frank, S., and Sima’an, K. (2016). ILLC-UvA Adaptation System (Scorpio) at WMT’16 IT-DOMAIN Task. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  32. Callison-Burch, C., Koehn, P., Monz, C., and Schroeder, J. (2009). Findings of the 2009 Workshop on Statistical Machine Translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 1–28, Athens, Greece. Association for Computational Linguistics. find
  33. Ding, S., Duh, K., Khayrallah, H., Koehn, P., and Post, M. (2016). The JHU Machine Translation Systems for WMT 2016. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  34. Callison-Burch, C., Koehn, P., Monz, C., and Zaidan, O. (2011). Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22–64, Edinburgh, Scotland. Association for Computational Linguistics. find
  35. Duma, M.-S. and Menzel, W. (2016). Data Selection for IT Texts using Paragraph Vector. In Proceedings of the First Conference on Machine G., Gomes, L., António Rodrigues, J. a., Neale, S., Silva, J. a., Querido, A., Rendeiro, N., and Branco, A. (2016). SMT and Hybrid systems of the QTLeap project i… find
  36. Translation, Berlin, Germany. Association for Computational Linguistics. find
  37. Durrani, N., Schmid, H., and Fraser, A. (2011). A joint sequence translation model with integrated reordering. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1045–1054. Association for Computational Linguistics. find
  38. Giménez, J. and Màrquez, L. (2010). Asiya: An Open Toolkit for Automatic Machine Translation (Meta-)Evaluation. The Prague Bulletin of Mathematical Linguistics, (94):77–86. find
  39. Dušek, O., Gomes, L., Novák, M., Popel, M., and Rosa, R. (2015). New Language Pairs in TectoMT. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 98– 104, Lisbon, Portugal. Association for Computational Linguistics. find
  40. Graham, Y. (2015). Improving Evaluation of Machine Translation Quality Estimation. In 53rd Annual Meeting of the Association for Computational Linguistics and Seventh International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, pages 1804–1813… find
  41. Dvorkovich, A., Gubanov, S., and Galinskaya, I. (2016). Yandex School of Data Analysis approach to English-Turkish translation at WMT16 News Translation Task. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  42. Graham, Y., Baldwin, T., Moffat, A., and Zobel, J. (2013). Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop & Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. find
  43. Esplà-Gomis, M., Sánchez-Martı́nez, F., and Forcada, M. (2015). UAlacant word-level machine translation quality estimation system at WMT 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 309– 315, Lisbon, Portugal. find
  44. Graham, Y., Baldwin, T., Moffat, A., and Zobel, J. (2014). Is machine translation getting better over time? In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 443–451, Gothenburg, Sweden. Association for Computational Linguistics. find
  45. Esplà-Gomis, M., Sánchez-Martı́nez, F., and Forcada, M. (2016). UAlacant word-level and phrase-level machine translation quality estimation systems at WMT 2016. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  46. Graham, Y., Baldwin, T., Moffat, A., and Zobel, J. (2016). Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, pages 1–28. find
  47. Federmann, C. (2012). Appraise: an open-source toolkit for manual evaluation of mt output. The Prague Bulletin of Mathematical Linguistics, 98:25–35. find
  48. Grönroos, S.-A., Virpioja, S., and Kurimo, M. (2016). Hybrid Morphological Segmentation for Phrase-Based Machine Translation. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  49. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  50. Gao, Q. and Vogel, S. (2008). Parallel Implementations of Word Alignment Tool. In Proceedings of the ACL 2008 Software Engineering, Testing, and Quality Assurance Workshop, pages 49–57, Columbus, Ohio. find
  51. Guillou, L., Hardmeier, C., Nakov, P., Stymne, S., Tiedemann, J., Versley, Y., Cettolo, M., Webber, B., and Popescu-Belis, A. (2016). Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction. In Proceedings of the First Conference on Machine Transla-Gaudio, R., Labaka, G., Agirre, E.… find
  52. Gwinnup, J., Anderson, T., Erdmann, G., Young, K., Kazi, M., Salesky, E., and Thompson, B. (2016). The AFRL-MITLL WMT16 News-Translation Task Systems. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  53. Junczys-Dowmunt, M., Dwojak, T., and Sennrich, R. (2016). The AMU-UEDIN Submission to the WMT16 News Translation Task: Attention-based NMT Models as Feature Functions in Phrase-based SMT. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Li… find
  54. Ha, T.-L., Cho, E., Niehues, J., Mediani, M., Sperber, M., Allauzen, A., and Waibel, A. (2016). The Karlsruhe Institute of Technology Systems for the News Translation Task in WMT 2016. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Lingu… find
  55. Junczys-Dowmunt, M. and Grundkiewicz, R. (2016). Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics. find
  56. Heafield, K. (2011). KenLM: Faster and Smaller Language Model Queries. In Proceedings of the EMNLP 2011 Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland, United Kingdom. find our card
  57. Kim, H. and Lee, J.-H. (2016). Recurrent Neural Network based Translation Quality Estimation. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  58. Herrmann, T., Niehues, J., and Waibel, A. (2013). Combining Word Reordering Methods on different Linguistic Abstraction Levels for Statistical Machine Translation. In Proceedings of the Seventh Workshop on Syntax, Semantics and Structure in Statistical Translation, Altanta, Georgia, USA. find
  59. Koehn, P. (2002). Europarl: A multilingual corpus for evaluation of machine translation. Unpublished, http://www.isi.edu/∼koehn/europarl/. ↗
  60. Koehn, P. (2004). Statistical Significance Tests for Machine Translation Evaluation. In Proceedings of EMNLP 2004, pages 388–395, Barcelona, Spain. find
  61. Hoang, C. and Sima’an, K. (2014). Latent domain translation models in mix-of-domains haystack. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1928–1939, Dublin, Ireland. Dublin City University and Association for Computational L… find
  62. Koehn, P. and Hoang, H. (2007). find
  63. Factored Translation Models. In EMNLP-CoNLL 2007, Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 868–876, Prague, Czech Republic. find
  64. Huck, M., Fraser, A., and Haddow, B. (2016). The Edinburgh/LMU Hierarchical Machine Translation System for WMT 2016. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  65. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the Asso… find
  66. Ive, J., Max, A., and Yvon, F. (2016). LIMSI’s Contribution to the WMT’16 Biomedical Translation Task. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  67. Jawaid, B., Kamran, A., Stanojević, M., and ojar, O. (2016). Results of the WMT16 Tuning Shared Task. In Proceedings of the First Con-Koehn, P. and Monz, C. (2006). Manual and automatic evaluation of machine translation be-Logacheva, V., Hokamp, C., and Specia, L. (2016b). MARMOT: A Toolkit for Tra… find
  68. Kozlova, A., Shmatova, M., and Frolov, A. (2016). YSDA Participation in the WMT’16 Quality Estimation Shared Task. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  69. Logacheva, V., Lukasik, M., and Specia, L. (2016c). Metrics for Evaluation of Word-Level Machine Translation Quality Estimation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany. find
  70. Kreutzer, J., Schamoni, S., and Riezler, S. (2015). QUality Estimation from ScraTCH (QUETCH): Deep Learning for Word-level Translation Quality Estimation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 297–303, Lisboa, Portugal. Association for Computational Linguisti… find
  71. Luong, N. Q., Besacier, L., and Lecouteux, B. (2014). Lig system for word level qe task at wmt14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 335– 341, Baltimore, Maryland, USA. Association for Computational Linguistics. find
  72. Landis, J. R. and Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33:159–174. find
  73. Mareček, D. (2016). Merged bilingual trees based on Universal Dependencies in Machine Translation. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find our card
  74. Liang, P., Taskar, B., and Klein, D. (2006). Alignment by agreement. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pages 104–111. Association for Computational Linguistics. find
  75. Martins, A., Almeida, M., and Smith, N. (2013). Turning on the turbo: Fast third-order non-projective turbo parsers. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 617–622, Sofia, Bulgaria. find our card
  76. Libovický, J., Helcl, J., Tlustý, M., Bojar, O., and Pecina, P. (2016). CUNI at Post-editing and Multimodal Translation Tasks. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics. find
  77. Martins, A. F. T., Astudillo, R., Hokamp, C., and Kepler, F. (2016). Unbabel’s Participation in the WMT16 Word-Level Translation Quality Estimation Shared Task. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  78. Lo, C.-k., Cherry, C., Foster, G., Stewart, D., Islam, R., Kazantseva, A., and Kuhn, R. (2016). NRC Russian-English Machine Translation System for WMT 2016. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  79. Mikolov, T., Yih, W.-t., and Zweig, G. (2013). Linguistic regularities in continuous space word representations. In Proceedings of NAACL 2013. find
  80. Logacheva, V., , and Specia, L. (2015). Phrase-level Quality Estimation for Machine Translation. In Proceedings of the 12th International Workshop on Spoken Language Translation, Da Nang, Vietnam. find
  81. Molchanov, A. and Bykov, F. (2016). PROMT Translation Systems for WMT 2016 Translation Tasks. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  82. Logacheva, V., Blain, F., and Specia, L. (2016a). USFD Phrase-level Quality Estimation Systems. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  83. Neves, M., Yepes, A. J., and Névéol, A. (2016). The Scielo Corpus: a Parallel Corpus of Scientific Publications for Biomedicine. In Proceedings of the Tenth International Conference M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., a… find
  84. Och, F. J. (2003). Minimum Error Rate Training in Statistical Machine Translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL ’03, pages 160–167, Sapporo, Japan. find
  85. Perez-de Viñaspre, O. and Labaka, G. (2016). IXA Biomedical Translation System at WMT16 Biomedical Translation Task. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  86. Och, F. J. and Ney, H. (2003). A Systematic Comparison of Various Statistical Alignment Models. Comput. Linguist., 29(1):19–51. find
  87. Okazaki, N. (2007). CRFsuite: a fast implementation of Conditional Random Fields. find
  88. Peter, J.-T., Alkhouli, T., Guta, A., and Ney, H. (2016a). The RWTH Aachen University English-Romanian Machine Translation System for WMT 2016. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  89. Paetzold, G. and Specia, L. (2016). SimpleNets: Quality Estimation with Resource-Light Neural Networks. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  90. Pahari, K., Kuila, A., Pal, S., Naskar, S. K., Bandyopadhyay, S., and van Genabith, J. (2016). JU-USAAR: A Domain Adaptive MT System. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  91. Peter, J.-T., Alkhouli, T., Ney, H., Huck, M., Braune, F., Fraser, A., Tamchyna, A., Bojar, O., Haddow, B., Sennrich, R., Blain, F., Specia, L., Niehues, J., Waibel, A., Allauzen, A., Aufrant, L., Burlot, F., knyazeva, e., Lavergne, T., Yvon, F., Daiber, J., and Pinnis, M. (2016b). The QT21/HimL Com… find
  92. Pal, S., Mihaela, V., Naskar, S. K., and van Genabith, J. (2015). USAAR-SAPE: An English– Spanish Statistical Automatic Post-Editing System. In Proceedings of the 10th Workshop on Statistical Machine Translation (WMT), pages 216–221. find
  93. Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press, Cambridge, Massachusetts. find
  94. Pal, S., Zampieri, M., and van Genabith, J. (2016). USAAR: An Operation Sequential Model for Automatic Statistical Post-Editing. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics. find
  95. Raybaud, S., Langlois, D., and Smali, K. (2011). this sentence is wrong. detecting errors in machine-translated sentences. Machine Translation, 25(1):1–34. find
  96. Rosa, R., Sudarikov, R., Novák, M., Popel, M., and Bojar, O. (2016). Dictionary-based Domain Adaptation of MT Systems without Retraining. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  97. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318. find our card
  98. Patel, R. N. and M, S. (2016). Translation Quality Estimation using Recurrent Neural Network. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  99. Sagemo, O. and Stymne, S. (2016). The UU Submission to the Machine Translation Quality Estimation Task. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  100. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, Sakaguchi, K., Post, M., and Van Durme, B. (2014). Efficient elicitation of annotations for Sennrich, R., Haddow, B., and Birch, A. (2016). Edinburgh Neural Machine Translation Systems for WMT 16. In Proceeding… find our card
  101. Sánchez-Cartagena, V. M. and Toral, A. (2016). Abu-MaTran at WMT 2016 Translation Task: Deep Learning, Morphological Segmentation and Tuning on Character Sequences. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  102. Shah, K., Bougares, F., Barrault, L., and Specia, L. (2016). SHEF-LIUM-NN: Sentence level Quality Estimation with Neural Network Features. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  103. Sim Smith, K., Aziz, W., and Specia, L. (2016). Cohere: A Toolkit for Local Coherence. In Proceedings of the 10th International Conference on Language Resources and Evaluation, Portorož, Slovenia. find
  104. Scarton, C., Beck, D., Shah, K., Sim Smith, K., and Specia, L. (2016). Word embeddings and discourse information for Quality Estimation. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  105. Simard, M., Goutte, C., and Isabelle, P. (2007). Statistical phrase-based post-editing. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 508–515, Rochester, New York. Associat… find
  106. Scarton, C. and Specia, L. (2014). Document-level translation quality estimation: exploring discourse and pseudo-references. In Proceedings of the 17th Annual Conference of the European Association for Machine Translation, pages 101–108, Dubrovnik, Croatia. find
  107. Scarton, C., Tan, L., and Specia, L. (2015a). USHEF and USAAR-USHEF participation in the WMT15 QE shared task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 317–322, Lisboa, Portugal. Association for Computational Linguistics. find
  108. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006). A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of Association for Machine Translation in the Americas, pages 223–231, Cambridge, Massachusetts, USA. find our card
  109. Scarton, C., Zampieri, M., Vela, M., van Genabith, J., and Specia, L. (2015b). Searching for Context: a Study on Document-Level Labels for Translation Quality Estimation. In Proceedings of the 18th Annual Conference of the European Association for Machine Translation, pages 121–128, Antalya, Turkey. find
  110. Specia, L., Frank, S., Sima’an, K., and Elliott, D. (2016). A Shared Task on Multimodal Machine Translation and Crosslingual Image Description. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find our card
  111. Specia, L., Paetzold, G., and Scarton, C. (2015). Multi-level Translation Quality Prediction with QuEst++. In 53rd Annual Meeting of the Association for Computational Linguistics and Seventh International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Pro… find
  112. Seddah, D., Kübler, S., and Tsarfaty, R. (2014). Introducing the SPMRL 2014 Shared Task on Parsing Morphologically-rich Languages. In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages, pages 103–109, Du… find
  113. Stahlberg, F., Hasler, E., and Byrne, B. (2016). The Edit Distance Transducer in Action: The University of Cambridge English-German System at WMT16. In Proceedings of the First Conference on Machine Translation, Berlin, Sennrich, R., Haddow, B., and Birch, A. (2015). Neural machine translation of ra… find
  114. Sudarikov, R., Popel, M., Bojar, O., Burchardt, A., and Klejch, O. (2016). Using MT-ComparEval. In Translation Evaluation: From Fragmented Tools and Data Sets to an Integrated Ecosystem, pages 76–82. find
  115. Tamchyna, A., Sudarikov, R., Bojar, O., and Fraser, A. (2016). CUNI-LMU Submissions in WMT2016: Chimera Constrained and Beaten. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  116. Tezcan, A., Hoste, V., and Macken, L. (2016). UGENT-LT3 SCATE Submission for WMT16 Shared Task on Quality Estimation. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  117. Tiedemann, J. (2009). News from OPUS - a collection of multilingual parallel corpora with tools and interfaces. In Proceedings of the Conference on Recent Advances in Natural Language Processing, pages 237–248. John Benjamins. find
  118. Tiedemann, J., Cap, F., Kanerva, J., Ginter, F., Stymne, S., Östling, R., and Weller-Di Marco, M. (2016). Phrase-Based SMT for Finnish with More Data, Better Models and Alternative Alignment and Translation Tools. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Assoc… find
  119. Williams, P., Sennrich, R., Nadejde, M., Huck, M., Haddow, B., and Bojar, O. (2016). Edinburgh’s Statistical Machine Translation Systems for WMT16. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  120. Wołk, K. and Marasek, K. (2016). PJAIT Systems for the WMT 2016. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics. find
  121. Yeh, A. (2000). More Accurate Tests for the Statistical Significance of Result Differences. In Coling-2000: the 18th Conference on Computational Linguistics, pages 947–953, Saarbrücken, Germany. find