Elamur Research
☰
home / papers / 2013 / Findings of the 2013 Workshop on Statistical Machine Translation

Findings of the 2013 Workshop on Statistical Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W13-2201.pdf

Summary

This report describes the shared tasks of the eighth WMT workshop, held at ACL 2013 in Sofia. It aimed to measure the current state of machine translation, release common benchmarks, and improve how translation quality is judged. The translation track covered ten directions between English and Czech, French, German, Spanish and Russian. Russian was new this year and attracted the most entries. Organizers supplied a 3,000-sentence news test set, professionally translated by Capita and Yandex. They also provided training corpora, including the new Common Crawl, Yandex 1M and Wiki Headlines data.

Submitted outputs were scored with automatic metrics and through a large human ranking campaign run in Appraise. That campaign combined judgments from task participants with paid Amazon Mechanical Turk workers. Collecting roughly ten times more judgments than before made most pairwise system comparisons statistically significant. Systems were therefore grouped into significance-based clusters rather than placed in a single total ranking.

A second-edition quality estimation task showed clear gains over the baseline. All data and judgments are public, making the collection a lasting resource for MT and evaluation research.

Key points

  • Translation task scale: 143 systems came from 23 institutions. Six anonymized external systems were added, three commercial and three online, and treated as unconstrained.
  • Test data: The test set drew on 52 news articles from Czech, English, French, German, Spanish and Russian outlets. Each was translated directly, without a pivot language.
  • Human evaluation: Researchers contributed about 280k pairwise rankings between May 17 and June 7. Turkers were paid $0.25 per HIT of three ranking tasks, with high-consensus control items embedded to check quality.
  • Quality estimation: The task had four subtasks (edit-rate prediction, ranking, post-editing time, word-level labels) on English-Spanish and German-English data. It drew 14 teams and 55 entries. SVR and Gaussian Processes performed strongly, and SHEF won the scoring variant of Task 1.1.
  • Progress over the previous year: In Task 1.1 scoring, 9 of 16 systems beat the baseline, compared with 5 of 20 the year before. The winner's relative improvement over the baseline rose from 13% to 19%. The metrics task is reported in a separate paper.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

205 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 186 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bojar2013findings,
  title = {{Findings of the 2013 Workshop on Statistical Machine Translation}},
  author = {Bojar, Ondřej and Buck, Christian and Callison-Burch, Chris and Federmann, Christian and Haddow, Barry and Koehn, Philipp and Monz, Christof and Post, Matt and Soricut, Radu and Specia, Lucia},
  booktitle = {Proceedings of the Eighth Workshop on Statistical Machine Translation},
  year = {2013},
  url = {https://aclanthology.org/W13-2201/}
}

References

95 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allauzen, A., Pécheux, N., Do, Q. K., Dinarelli, M., Lavergne, T., Max, A., Le, H.-S., and Yvon, F. (2013). LIMSI @ WMT13. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 60–67, Sofia, Bulgaria. Association for Computational Linguistics. find
  2. Almaghout, H. and Specia, L. (2013). A CCG-based Quality Estimation Metric for Statistical Machine Translation. In Proceedings of MT Summit XIV (to appear), Nice, France. find
  3. Ammar, W., Chahuneau, V., Denkowski, M., Hanneman, G., Ling, W., Matthews, A., Murray, http://statmt.org/wmt13/results.html ↗
  4. K., Segall, N., Lavie, A., and Dyer, C. (2013). The CMU machine translation systems at WMT 2013: Syntax, synthetic translation options, and pseudo-references. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 68–75, Sofia, Bulgaria. Association for Computational Linguis… find
  5. Avramidis, E. (2012). Comparative Quality Estimation: Automatic Sentence-Level Ranking of Multiple Machine Translation Outputs. In Proceedings of 24th International Conference on Computational Linguistics, pages 115–132, Mumbai, India. find
  6. Avramidis, E. and Popovic, M. (2013). Selecting feature sets for comparative and time-oriented quality estimation of machine translation output. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 327– 334, Sofia, Bulgaria. Association for Computational Linguistics. find
  7. Avramidis, E., Popović, M., Vilar, D., and Burchardt, A. (2011). Evaluate with confidence estimation: Machine ranking of translation outputs using grammatical features. In Proceedings of the Sixth Workshop on Statistical Machine Translation. find
  8. Aziz, W., Mitkov, R., and Specia, L. (2013). Ranking Machine Translation Systems via Post-Editing. In Proc. of Text, Speech and Dialogue (TSD), Lecture Notes in Artificial Intelligence, Berlin / Heidelberg. Západočeská univerzita v Plzni, Springer Verlag. find
  9. Bach, N., Huang, F., and Al-Onaizan, Y. (2011). Goodness: A method for measuring machine translation confidence. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 211–219, Portland, Oregon, USA. find
  10. Beck, D., Shah, K., Cohn, T., and Specia, L. (2013). SHEF-Lite: When less is more for translation quality estimation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 335–340, Sofia, Bulgaria. Association for Computational Linguistics. find
  11. Biçici, E., Groves, D., and van Genabith, J. (2013). Predicting sentence translation quality using extrinsic and language independent features. Machine Translation. find
  12. Biçici, E. and van Genabith, J. (2013). CNGL-CORE: Referential translation machines for measuring semantic similarity. In *SEM 2013: The Second Joint Conference on Lexical and Computational Semantics, Atlanta, Georgia, USA. Association for Computational Linguistics. find
  13. Bicici, E. (2013a). Feature decay algorithms for fast deployment of accurate statistical machine translation systems. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 76–82, Sofia, Bulgaria. Association for Computational Linguistics. find
  14. Bicici, E. (2013b). Referential translation machines for quality estimation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 341–349, Sofia, Bulgaria. Association for Computational Linguistics. find
  15. Bı́lek, K. and Zeman, D. (2013). CUni multilingual matrix in the WMT 2013 shared task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 83–89, Sofia, Bulgaria. Association for Computational Linguistics. find
  16. Bojar, O., Kos, K., and Mareček, D. (2010). Tackling Sparse Data Issue in Machine Translation Evaluation. In Proceedings of the ACL 2010 Conference Short Papers, pages 86–91, Uppsala, Sweden. Association for Computational Linguistics. find
  17. Bojar, O., Rosa, R., and Tamchyna, A. (2013). Chimera – three heads for English-to-Czech translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 90–96, Sofia, Bulgaria. Association for Computational Linguistics. find
  18. Borisov, A., Dlougach, J., and Galinskaya, I. (2013). Yandex school of data analysis machine translation systems for WMT13. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 97–101, Sofia, Bulgaria. Association for Computational Linguistics. find
  19. Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., and Schroeder, J. (2007). (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation (WMT07), Prague, Czech Republic. find
  20. Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., and Schroeder, J. (2008). Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation (WMT08), Colmbus, Ohio. find
  21. Callison-Burch, C., Koehn, P., Monz, C., Peterson, K., Przybocki, M., and Zaidan, O. F. (2010). Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT10), Uppsala, Swede… find
  22. Callison-Burch, C., Koehn, P., Monz, C., Post, M., Soricut, R., and Specia, L. (2012). Findings of the 2012 workshop on statistical machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10– 51, Montréal, Canada. Association for Computational Linguisti… find our card
  23. Callison-Burch, C., Koehn, P., Monz, C., and Schroeder, J. (2009). Findings of the 2009 workshop on statistical machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT09), Athens, Greece. find
  24. Callison-Burch, C., Koehn, P., Monz, C., and Zaidan, O. (2011). Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22–64, Edinburgh, Scotland. find
  25. Camargo de Souza, J. G., Buck, C., Turchi, M., and Negri, M. (2013). FBK-UEdin participation to the WMT13 quality estimation shared task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 350– 356, Sofia, Bulgaria. Association for Computational Linguistics. find
  26. Charniak, E. and Johnson, M. (2005). Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 173–180. Association for Computational Linguistics. find our card
  27. Cho, E., Ha, T.-L., Mediani, M., Niehues, J., Herrmann, T., Slawik, I., and Waibel, A. (2013). The Karlsruhe Institute of Technology translation systems for the WMT 2013. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 102–106, Sofia, Bulgaria. Association for Computa… find
  28. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurment, 20(1):37–46. find
  29. Cohn, T. and Specia, L. (2013). Modelling Annotator Bias with Multi-task Gaussian Processes: An Application to Machine Translation Quality Estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (to appear). find
  30. Collins, M. (2002). Discriminative training methods for hidden markov models: theory and experiments with perceptron algorithms. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing - Volume 10, EMNLP ’02, pages 1–8, Stroudsburg, PA, USA. Association for Comput… find our card
  31. Denkowski, M. and Lavie, A. (2010). Meteor-next and the meteor paraphrase tables: improved evaluation support for five target languages. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, WMT ’10, pages 339–342, Stroudsburg, PA, USA. Association for Comput… find
  32. Durgar El-Kahlout, I. and Mermer, C. (2013). TÜbtak-blgem german-english machine translation systems for w13. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 107–111, Sofia, Bulgaria. Association for Computational Linguistics. find
  33. Durrani, N., Fraser, A., Schmid, H., Sajjad, H., and Farkas, R. (2013a). Munich-Edinburgh-Stuttgart submissions of OSM systems at WMT13. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 120–125, Sofia, Bulgaria. Association for Computational Linguistics. find
  34. Durrani, N., Haddow, B., Heafield, K., and Koehn, P. (2013b). Edinburgh’s machine translation systems for European language pairs. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 112–119, Sofia, Bulgaria. Association for Computational Linguistics. find
  35. Eidelman, V., Wu, K., Ture, F., Resnik, P., and Lin, J. (2013). Towards efficient large-scale feature-rich statistical machine translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 126–131, Sofia, Bulgaria. Association for Computational Linguistics. find
  36. Federmann, C. (2012). Appraise: An Open-Source Toolkit for Manual Evaluation of Machine Translation Output. The Prague Bulletin of Mathematical Linguistics (PBML), 98:25– 35. find
  37. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. find
  38. Formiga, L., Costa-jussà, M. R., Mariño, J. B., Fonollosa, J. A. R., Barrón-Cedeño, A., and Marquez, L. (2013a). The TALP-UPC phrase-based translation systems for WMT13: System combination with morphology generation, domain adaptation and corpus filtering. In Proceedings of the Eighth Workshop o… find
  39. Formiga, L., Gonzàlez, M., Barrón-Cedeño, A., Fonollosa, J. A. R., and Marquez, L. (2013b). The TALP-UPC approach to system selection: Asiya features and pairwise classification using random forests. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 357–362, Sofia, B… find
  40. Formiga, L., Màrquez, L., and Pujantell, J. (2013c). Real-life translation quality estimation for mt system selection. In Proceedings of MT Summit XIV (to appear), Nice, France. find
  41. Galuščáková, P., Popel, M., and Bojar, O. (2013). PhraseFix: Statistical post-editing of TectoMT. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 139–145, Sofia, Bulgaria. Association for Computational Linguistics. find
  42. Gispert, A., Blackwood, G., Iglesias, G., and Byrne, W. (2013). N-gram posterior probability confidence measures for statistical machine translation: an empirical study. Machine Translation, 27:85–114. find
  43. Gonzàlez, M., Giménez, J., and Màrquez, L. (2012). A graphical interface for mt evaluation and error analysis. In Proceedings of the ACL 2012 System Demonstrations, pages 139–144, Jeju Island, Korea. find
  44. Green, S., Cer, D., Reschke, K., Voigt, R., Bauer, J., Wang, S., Silveira, N., Neidert, J., and Manning, C. D. (2013). Feature-rich phrase-based translation: Stanford University’s submission to the WMT 2013 translation task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, p… find
  45. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., and Witten, I. H. (2009). The weka data mining software: an update. SIGKDD Explor. Newsl., 11(1):10–18. find
  46. Han, A. L.-F., Wong, D. F., Chao, L. S., Lu, Y., He, L., Wang, Y., and Zhou, J. (2013). A description of tunable machine translation evaluation systems in WMT13 metrics task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 412–419, Sofia, Bulgaria. Association for Com… find
  47. Hildebrand, S. and Vogel, S. (2013). MT quality estimation: The CMU system for WMT’13. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 371–377, Sofia, Bulgaria. Association for Computational Linguistics. find
  48. Hosmer, D. (1989). Applied logistic regression. Wiley, New York, 8th edition. find
  49. Huet, S., Manishina, E., and Lefèvre, F. (2013). Factored machine translation systems for Russian-English. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 152–155, Sofia, Bulgaria. Association for Computational Linguistics. find
  50. Hunt, E., Martin, J., and Stone, P. (1966). Experiments in Induction. Academic Press, New York. find
  51. Kaplan, R., Riezler, S., King, T., Maxwell, J., Vasserman, A., and Crouch, R. (2004). Speed and accuracy in shallow and deep stochastic parsing. In Proceedings of the Human Language Technology Conference and the 4th Annual Meeting of the North American Chapter of the Association for Computational Li… find
  52. Kauchak, D. and Barzilay, R. (2006). Paraphrasing for automatic evaluation. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, HLT-NAACL ’06, pages 455–462, Stroudsburg, PA, USA. Association for… find
  53. Koehn, P. (2012). Simulating human judgment in machine translation evaluation campaigns. In International Workshop on Spoken Language Translation (IWSLT). find
  54. Koehn, P. and Monz, C. (2006). Manual and automatic evaluation of machine translation between European languages. In Proceedings of NAACL 2006 Workshop on Statistical Machine Translation, New York, New York. find
  55. Kononenko, I. (1994). Estimating attributes: analysis and extensions of RELIEF. In Proceedings of the European conference on machine learning on Machine Learning, pages 171–182, Secaucus, NJ, USA. Springer-Verlag New York, Inc. find
  56. Kos, K. and Bojar, O. (2009). Evaluation of Machine Translation Metrics for Czech as the Target Language. Prague Bulletin of Mathematical Linguistics, 92:135–147. find
  57. Landis, J. R. and Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33:159–174. find
  58. Langlois, D., Raybaud, S., and Smaı̈li, K. (2012). Loria system for the wmt12 quality estimation shared task. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 114–119, Montréal, Canada. find
  59. Langlois, D. and Smaili, K. (2013). LORIA system for the WMT13 quality estimation shared task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 378– 383, Sofia, Bulgaria. Association for Computational Linguistics. find
  60. Le, H. S., Oparin, I., Allauzen, A., Gauvain, J.-L., and Yvon, F. (2011). Structured output layer neural network language model. In ICASSP, pages 5524–5527. find
  61. Lin, C.-J., Weng, R. C., and Keerthi, S. S. (2007). Trust region Newton methods for large-scale logistic regression. In Proceedings of the 24th international conference on Machine learning - ICML ’07, pages 561–568, New York, New York, USA. ACM Press. find
  62. Luong, N. Q., Lecouteux, B., and Besacier, L. (2013). LIG system for WMT13 QE task: Investigating the usefulness of features in word confidence estimation for MT. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 384–389, Sofia, Bulgaria. Association for Computational L… find
  63. Macháček, M. and Bojar, O. (2013). Results of the WMT13 metrics shared task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 43–49, Sofia, Bulgaria. Association for Computational Linguistics. find
  64. Matusov, E. and Leusch, G. (2013). Omnifluent English-to-French and Russian-to-English systems for the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 156–161, Sofia, Bulgaria. Association for Computational Linguistics… find
  65. McCallum, A. K. (2002). MALLET: a machine learning for language toolkit. find
  66. Miceli Barone, A. V. and Attardi, G. (2013). Pre-reordering for machine translation using transition-based walks on dependency parse trees. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 162– 167, Sofia, Bulgaria. Association for Computational Linguistics. find
  67. Moreau, E. and Rubino, R. (2013). An approach using style classification features for quality estimation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 427–432, Sofia, Bulgaria. Association for Computational Linguistics. find
  68. Nadejde, M., Williams, P., and Koehn, P. (2013). Edinburgh’s syntax-based machine translation systems. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 168–174, Sofia, Bulgaria. Association for Computational Linguistics. find
  69. Okita, T., Liu, Q., and van Genabith, J. (2013). Shallow semantically-informed PBSMT and HPBSMT. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 175–182, Sofia, Bulgaria. Association for Computational Linguistics. find
  70. Özgür, A., Özgür, L., and Güngör, T. (2005). Text categorization with class-based and corpus-based keyword selection. In Proceedings of the 20th International Conference on Computer and Information Sciences, ISCIS’05, pages 606–615, Berlin, Heidelberg. Springer. find
  71. Peitz, S., Mansour, S., Huck, M., Freitag, M., Ney, H., Cho, E., Herrmann, T., Mediani, M., Niehues, J., Waibel, A., Allauzen, A., Khanh Do, Q., Buschbeck, B., and Wandmacher, T. (2013a). find
  72. Joint WMT 2013 submission of the QUAERO project. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 183–190, Sofia, Bulgaria. Association for Computational Linguistics. find
  73. Peitz, S., Mansour, S., Peter, J.-T., Schmidt, C., Wuebker, J., Huck, M., Freitag, M., and Ney, H. (2013b). The RWTH aachen machine translation system for WMT 2013. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 191–197, Sofia, Bulgaria. Association for Computational… find
  74. Pighin, D., Formiga, L., and Màrquez, L. (2012). A graph-based strategy to streamline translation quality assessments. In Proceedings of the Tenth Conference of the Association for Machine Translation in the Americas (AMTA’2012), San Diego, USA. find
  75. Pino, J., Waite, A., Xiao, T., de Gispert, A., Flego, F., and Byrne, W. (2013). The University of Cambridge Russian-English system at WMT13. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 198–203, Sofia, Bulgaria. Association for Computational Linguistics. find
  76. Post, M., Ganitkevitch, J., Orland, L., Weese, J., Cao, Y., and Callison-Burch, C. (2013). Joshua 5.0: Sparser, better, faster, server. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 204–210, Sofia, Bulgaria. Association for Computational Linguistics. find
  77. Rubino, R., Toral, A., Cortés Vaı́llo, S., Xie, J., Wu, X., Doherty, S., and Liu, Q. (2013a). The CNGL-DCU-Prompsit translation systems for WMT13. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 211–216, Sofia, Bulgaria. Association for Computational Linguistics. find
  78. Rubino, R., Wagner, J., Foster, J., Roturier, J., Samad Zadeh Kaljahi, R., and Hollowood, F. (2013b). DCU-Symantec at the WMT 2013 quality estimation shared task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 390–395, Sofia, Bulgaria. Association for Computational L… find
  79. Sajjad, H., Smekalova, S., Durrani, N., Fraser, A., and Schmid, H. (2013). QCRI-MES submission at WMT13: Using transliteration mining to improve statistical machine translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 217–222, Sofia, Bulgaria. Association for… find
  80. Schmidt, H. (1994). Probabilistic part-of-speech tagging using decision trees. In Proceedings of the International Conference on New Methods in Natural Language Processing. find
  81. Seginer, Y. (2007). Learning Syntactic Structure. PhD thesis, University of Amsterdam. find
  82. Shah, K., Cohn, T., and Specia, L. (2013). An Investigation on the Effectiveness of Features for Translation Quality Estimation. In Proceedings of MT Summit XIV (to appear), Nice, France. find
  83. Singh, A. K., Wisniewski, G., and Yvon, F. (2013). LIMSI submission for the WMT’13 quality estimation task: an experiment with ngram posteriors. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 396–402, Sofia, Bulgaria. Association for Computational Linguistics. find
  84. Smith, J., Saint-Amand, H., Plamada, M., Koehn, P., Callison-Burch, C., and Lopez, A. (2013). Dirt cheap web-scale parallel text from the Common Crawl. In Proceedings of the 2013 Conference of the Association for Computational Linguistics (ACL 2013), Sofia, Bulgaria. Association for Computational Li… find
  85. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Biennial Conference of the Association for Machine Translation in the Americas (AMTA-2006), Cambridge, Massachusetts. find our card
  86. Snover, M., Madnani, N., Dorr, B. J., and Schwartz, R. (2009). Fluency, adequacy, or hter?: exploring different human judgments with a tunable mt metric. In Proceedings of the Fourth Workshop on Statistical Machine Translation, StatMT ’09, pages 259–268, Stroudsburg, PA, USA. Association for Computa… find
  87. Soricut, R., Bach, N., and Wang, Z. (2012). The SDL Language Weaver Systems in the WMT12 Quality Estimation Shared Task. In Proceedings of the 7th Workshop on Statistical Machine Translation, pages 145–151. find
  88. Souza, J. G. C. d., Espl-Gomis, M., Turchi, M., and Negri, M. (2013). Exploiting qualitative information from automatic word alignment for cross-lingual nlp tasks. In The 51st Annual Meeting of the Association for Computational Linguistics - Short Papers (ACL Short Papers 2013). find
  89. Specia, L. (2011). Exploiting Objective Annotations for Measuring Translation Post-editing Effort. In Proceedings of the 15th Conference of the European Association for Machine Translation, pages 73–80, Leuven. find
  90. Specia, L., Shah, K., de Souza, J. G. C., and Cohn, T. (2013). QuEst - A Translation Quality Estimation Framework. In Proceedings of the 51th Conference of the Association for Computational Linguistics (ACL), Demo Session, Sofia, Bulgaria. Association for Computational Linguistics. find
  91. Stolcke, A. (2002). SRILM – An extensible language modeling toolkit. In Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP 2002), pages 901– 904. find
  92. Stone, M. and Brooks, R. J. (1990). Continuum regression: cross-validated sequentially constructed prediction embracing ordinary least squares, partial least squares and principal components regression. Journal of the Royal Statistical Society Series B Methodological, 52(2):237–269. find
  93. Stymne, S., Hardmeier, C., Tiedemann, J., and Nivre, J. (2013). Tunable distortion limits and corpus cleaning for SMT. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 223–229, Sofia, Bulgaria. Association for Computational Linguistics. find
  94. Tantug, A. C., Oflazer, K., and El-Kahlout, I. D. (2008). BLEU+: a Tool for Fine-Grained BLEU Computation. In (ELRA), E. L. R. A., editor, Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), Marrakech, Morocco. find
  95. Weller, M., Kisselew, M., Smekalova, S., Fraser, A., Schmid, H., Durrani, N., Sajjad, H., and Farkas, R. (2013). Munich-Edinburgh-Stuttgart submissions at WMT13: Morphological and syntactic processing for SMT. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 230–237, S… find