Elamur Research⌕
☰
home / papers / 2017 / Discourse Structure in Machine Translation Evaluation

Discourse Structure in Machine Translation Evaluation

Summary

Standard metrics for machine translation evaluation, such as BLEU, mostly compare surface word overlap. They ignore how clauses within a sentence relate to one another. Joty and colleagues test whether sentence-level discourse structure can close part of this gap. They parse both system outputs and references into RST trees using the publicly available parser of Joty, Carenini, and Ng (2015). They then measure tree similarity with the all-subtree kernel of Collins and Duffy, which yields the DR-LEX family of measures.

On the WMT11 and WMT12 metrics benchmarks, adding DR-LEX to existing metrics usually raises correlation with human judgments, even with uniform weights. Learning the weights from human pairwise rankings improves results further. These ideas fed into DISCOTKparty, the top metric at WMT14.

An ablation shows that every layer of the tree contributes, and nuclearity matters more than relation labels. The findings indicate that discourse provides signal beyond lexical and syntactic metrics. This motivates document-level evaluation and discourse-aware MT.

Key points

  • The core measure, DR-LEX, scores how many subtrees a hypothesis RST tree shares with the reference tree. Variants differ in the tree representation used, for example with or without lexical leaves.
  • Weights for combining DR-LEX with metrics like BLEU, TER and METEOR were learned from human judgments. This gave average relative gains of 22–35% in segment-level correlation. The models stayed stable across WMT years and source languages.
  • DISCOTKparty combines 23 metrics: five DR-LEX variants plus strong metrics from the ASIYA toolkit. It ranked first at the WMT14 Metrics task at both the system and segment levels. Within it, DR-LEX1 received the fourth-highest weight, comparable to BLEU.
  • The ablation found that hierarchy, relation types, and nuclearity each add value, with nuclearity the most informative. Human-preferred translations had RST trees consistently closer to the reference across all language pairs.
  • The main limitation is that the work covers only sentence-level parsing, since no document-level human quality annotations exist. The authors propose discourse-based re-ranking and learning from fine-grained tree features as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Discourse analysis · Machine translation

Cited by

15 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 13 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00298

@article{joty2017discourse,
  title = {{Discourse Structure in Machine Translation Evaluation}},
  author = {Joty, Shafiq and Guzmán, Francisco and Màrquez, Lluís and Nakov, Preslav},
  journal = {Computational Linguistics},
  year = {2017},
  doi = {10.1162/coli\_a\_00298},
  url = {https://doi.org/10.1162/coli_a_00298}
}

References

121 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asher, Nicholas and Alex Lascarides. 2003. Logics of Conversation. Cambridge University Press. find
  2. Bazrafshan, Marzieh and Daniel Gildea. 2014. Comparing representations of semantic roles for string-to-tree decoding. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1786–1791, Doha. find
  3. Becher, Viktor. 2011. When and why do translators add connectives? A corpus-based study. Target. International Journal of Translation Studies, 23(1):26–47. ↗
  4. Ben, Guosheng, Deyi Xiong, Zhiyang Teng, Yajuan Lü, and Qun Liu. 2013. Bilingual lexical cohesion trigger model for document-level machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 382–386, Sofia. find
  5. Bojar, Ondřej, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12–58, Baltimore, MD. find our card
  6. Bojar, Ondřej, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 metrics shared task. In Proceedings of the First Conference on Machine Translation, pages 199–231, Berlin. find our card
  7. Brown, Peter F., Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. find
  8. Callison-Burch, Chris, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 136–158, Prague. find our card
  9. Callison-Burch, Chris, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 70–106, Columbus, OH. find our card
  10. Callison-Burch, Chris, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 17–53, Uppsala. find our card
  11. Callison-Burch, Chris, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 workshop on statistical machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10–51, Montréal. find our card
  12. Callison-Burch, Chris, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009. Findings of the 2009 workshop on statistical machine translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 1–28, Athens. find our card
  13. Callison-Burch, Chris, Philipp Koehn, Christof Monz, and Omar Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 22–64, Edinburgh. find our card
  14. Callison-Burch, Chris, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of BLEU in machine translation research. In Proceedings of 11th Conference of the European Chapter of the Association for Computational Linguistics, pages 249–256, Trento. find
  15. Carlson, Lynn and Daniel Marcu. 2001, Discourse tagging reference manual. Technical Report ISI-TR-545, University of Southern California Information Sciences Institute. find
  16. Carpuat, Marine. 2009. One translation per discourse. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions (SEW-2009), pages 19–27, Boulder, CO. find
  17. Carpuat, Marine and Michel Simard. 2012. The trouble with SMT consistency. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 442–449, Montreal. find
  18. Carpuat, Marine and Dekai Wu. 2007. Improving statistical machine translation using word sense disambiguation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 61–72, Prague. find
  19. Cartoni, Bruno, Sandrine Zufferey, Thomas Meyer, and Andrei Popescu-Belis. 2011. How comparable are parallel corpora? Measuring the distribution of general vocabulary and connectives. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web, pages 78–86, Portland, OR. find
  20. Chan, Yee Seng, Hwee Tou Ng, and David Chiang. 2007. Word sense disambiguation improves statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 33–40, Prague. find
  21. Chiang, David. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 263–270, Ann Arbor, MI. find our card
  22. Chiang, David, Yuval Marton, and Philip Resnik. 2008. Online large-margin training of syntactic and structural translation features. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Honolulu, HI. find
  23. Collins, Michael and Nigel Duffy. 2001. Convolution kernels for natural language. In Neural Information Processing Systems, pages 625–632, Vancouver. ↗
  24. Coughlin, Deborah. 2003. Correlating automated and human assessments of machine translation quality. In Proceedings of Machine Translation Summit IX, pages 23–27, New Orleans, LA. find
  25. Culy, Christopher and Susanne Z. Riehemann. 2003. The limits of n-gram translation evaluation metrics. In Proceedings of Machine Translation Summit IX, pages 1–8, New Orleans, LA. find
  26. Danlos, Laurence. 2009. D-STAG: A discourse analysis formalism based on synchronous tags. TAL, 50(1):111–143. find
  27. Denkowski, Michael and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 85–91, Edinburgh. find our card
  28. Denkowski, Michael and Alon Lavie. 2012. Challenges in predicting machine translation utility for human post-editors. In Proceedings of the Tenth Conference of the Association for Machine Translation in the Americas, San Diego, CA. find
  29. Doddington, George. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, pages 138–145, Morgan Kaufmann Publishers Inc., San Diego, CA. find
  30. Galley, Michel, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What’s in a translation rule?. In Proceedings of the 2004 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technology, pages 273–280, San Diego, CA. ↗ our card
  31. Giménez, Jesús and Lluı́s Màrquez. 2007. Linguistic features for automatic evaluation of heterogenous MT systems. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 256–264, Prague. find
  32. Giménez, Jesús and Lluı́s Màrquez. 2009. On the robustness of syntactic and semantic features for automatic MT evaluation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 250–258, Athens. find
  33. Giménez, Jesús and Lluı́s Màrquez. 2010a. Asiya: An open toolkit for automatic machine translation (meta-)evaluation. The Prague Bulletin of Mathematical Linguistics, 94:77–86. ↗
  34. Giménez, Jesús and Lluı́s Màrquez. 2010b. Linguistic measures for automatic machine translation evaluation. Machine Translation, 24(1):77–86. ↗
  35. Giménez, Jesús, Lluı́s Màrquez, Elisabet Comelles, Irene Castellón, and Victoria Arranz. 2010. Document-level automatic MT evaluation based on discourse representations. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 333–338, Uppsala. find
  36. Gong, Zhengxian, Min Zhang, and Guodong Zhou. 2011. Cache-based document-level statistical machine translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 909–919, Edinburgh. find
  37. Gonzàlez, Meritxell, Jesús Giménez, and Lluı́s Màrquez. 2012. A graphical interface for MT evaluation and error analysis. In Proceedings of the ACL 2012 System Demonstrations, pages 139–144, Jeju Island. find
  38. Guillou, Liane. 2012. Improving pronoun translation for statistical machine translation. In Proceedings of the Student Research Workshop at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 1–10, Avignon. find
  39. Guillou, Liane. 2013. Analysing lexical consistency in translation. In Proceedings of the Workshop on Discourse in Machine Translation, pages 10–18, Sofia. find
  40. Guillou, Liane, Christian Hardmeier, Preslav Nakov, Sara Stymne, Jörg Tiedemann, Yannick Versley, Mauro Cettolo, Bonnie Webber, and Andrei Popescu-Belis. 2016. Findings of the 2016 WMT shared task on cross-lingual pronoun prediction. In Proceedings of the First Conference on Machine Translation, pages 525–542, Berlin. find
  41. Guzmán, Francisco, Shafiq Joty, Lluı́s Màrquez, Alessandro Moschitti, Preslav Nakov, and Massimo Nicosia. 2014a. Learning to differentiate better from worse translations. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 214–220, Doha. find
  42. Guzmán, Francisco, Shafiq Joty, Lluı́s Màrquez, and Preslav Nakov. 2014b. Using discourse structure improves machine translation evaluation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 687–698, Baltimore, MD. find
  43. Guzmán, Francisco, Shafiq Joty, Lluı́s Màrquez, and Preslav Nakov. 2015. Pairwise neural machine translation evaluation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 805–814, Beijing. find
  44. Hajlaoui, Najeh. 2013. Are ACT’s scores increasing with better translation quality? In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 408–413, Sofia. find
  45. Hajlaoui, Najeh and Andrei Popescu-Belis. 2012. Translating English discourse connectives into Arabic: a corpus-based analysis and an evaluation metric. In Fourth Workshop on Computational Approaches to Arabic Script-based Languages at Proceedings of the Tenth Biennial Conference of the Association for Machine Translation in the Americas. find
  46. Halliday, Michael and Ruqaiya Hasan. 1976. Cohesion in English. Longman, London. find
  47. Hardmeier, Christian. 2012. Discourse in statistical machine translation. A survey and a case study. Discours. Revue de linguistique, psycholinguistique et informatique, 11(8726). ↗
  48. Hardmeier, Christian. 2014. Discourse in Statistical Machine Translation. Ph.D. thesis, Uppsala University, Uppsala, Sweden. find
  49. Hardmeier, Christian and Marcello Federico. 2010. Modelling pronominal anaphora in statistical machine translation. In Proceedings of the International Workshop on Spoken Language Translation, pages 283–289, Paris. find our card
  50. Hardmeier, Christian, Preslav Nakov, Sara Stymne, Jörg Tiedemann, Yannick Versley, and Mauro Cettolo. 2015. Pronoun-focused MT and cross-lingual pronoun prediction: Findings of the 2015 DiscoMT shared task on pronoun translation. In Proceedings of the EMNLP 2015 Workshop on Discourse in Machine Translation, Lisbon. find
  51. Hardmeier, Christian, Joakim Nivre, and Jörg Tiedemann. 2012. Document-wide decoding for phrase-based statistical machine translation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1179–1190, Jeju Island. find
  52. Hobbs, Jerry. 1979. Coherence and Coreference. Cognitive Science, 3:67–90. ↗
  53. Hopkins, Mark and Jonathan May. 2011. Tuning as ranking. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Edinburgh. find
  54. Hovy, Eduard, Margaret King, and Andrei Popescu-Belis. 2002. Principles of context-based machine translation evaluation. Machine Translation, 17(3–4):43–75. ↗
  55. Joty, Shafiq, Giuseppe Carenini, and Raymond T. Ng. 2012. A novel discriminative framework for sentence-level discourse analysis. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 904–915, Jeju Island. find
  56. Joty, Shafiq, Giuseppe Carenini, and Raymond T. Ng. 2015. CODRA: A Novel Discriminative Framework for Rhetorical Analysis. Computational Linguistics, 41(3):385–435. ↗ our card
  57. Joty, Shafiq, Francisco Guzmán, Lluı́s Màrquez, and Preslav Nakov. 2014. DiscoTK: Using discourse structure for machine translation evaluation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 402–408, Baltimore, MD. find
  58. Kamp, Hans and Uwe Reyle. 1993. From Discourse to Logic: Introduction to Model Theoretic Semantics of Natural Language, Formal Logic and Discourse Representation Theory. 42. Kluwer Academic Publishers. find
  59. Kendall, Maurice. 1938. A new measure of rank correlation. Biometrika, 1–2(30):81–89. ↗
  60. Koehn, Philipp, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, pages 48–54, Edmonton, Canada. ↗
  61. Kullback, Solomon and Richard A. Leibler. 1951. On information and sufficiency. Annals of Mathematical Statistics, 22(1):79–86. ↗
  62. Lavie, Alon and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228–231, Prague. find our card
  63. Lavie, Alon and Michael J. Denkowski. 2009. The METEOR metric for automatic evaluation of machine translation. Machine Translation, 23:105–115. ↗
  64. Le Nagard, Ronan and Philipp Koehn. 2010. Aiding pronoun translation with co-reference resolution. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 252–261, Uppsala. find our card
  65. Li, Jiwei, Rumeng Li, and Eduard Hovy. 2014. Recursive deep models for discourse parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2061–2069, Doha. find our card
  66. Li, Junyi Jessy, Marine Carpuat, and Ani Nenkova. 2014. Assessing the discourse factors that influence the quality of machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 283–288, Baltimore, MD. find
  67. Lin, Chin-Yew. 2004. ROUGE: A package for automatic evaluation of summaries. In Proceedings of Workshop on Text Summarization Branches Out, pages 74–81, Barcelona. find
  68. Liu, Ding and Daniel Gildea. 2005. Syntactic features for evaluation of machine translation. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 25–32, Ann Arbor, MI. find our card
  69. Lo, Chi kiu, Anand Karthik Tumuluru, and Dekai Wu. 2012. Fully automatic semantic MT evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 243–252, Montréal. find
  70. Lo, Chi kiu and Dekai Wu. 2011. Meant: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 220–229, Portland, OR. find our card
  71. Louis, Annie and Bonnie Webber. 2014. Structured and unstructured cache models for SMT domain adaptation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 155–163, Gothenburg. find
  72. Macháček, Matouš and Ondřej Bojar. 2013. Results of the WMT13 metrics shared task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia. find
  73. Macháček, Matouš and Ondřej Bojar. 2014. Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, MD. find our card
  74. Mann, William and Sandra Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3):243–281. ↗ our card
  75. Marcu, Daniel, Lynn Carlson, and Maki Watanabe. 2000. The automatic translation of discourse structures. In Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, pages 9–17, Seattle, WA. find
  76. Meyer, Thomas. 2011. Disambiguating temporal-contrastive connectives for machine translation. In Proceedings of the ACL 2011 Student Session, pages 46–51, Portland, OR. find
  77. Meyer, Thomas. 2014. Discourse-level Features for Statistical Machine Translation. Ph.D. thesis, École polytechnique Fédérale de Lausanne (EPFL), Lausanne. ↗
  78. Meyer, Thomas and Lucie Poláková. 2013. Machine translation with many manually labeled discourse connectives. In Proceedings of the Workshop on Discourse in Machine Translation, pages 43–50, Sofia. find our card
  79. Meyer, Thomas and Andrei Popescu-Belis. 2012. Using sense-labeled discourse connectives for statistical machine translation. In Proceedings of the Joint Workshop on Exploiting Synergies between Information Retrieval and Machine Translation (ESIRMT) and Hybrid Approaches to Machine Translation (HyTra), pages 129–138, Avignon. find our card
  80. Meyer, Thomas, Andrei Popescu-Belis, Najeh Hajlaoui, and Andrea Gesmundo. 2012. Machine translation of labeled discourse connectives. In Proceedings of the Tenth Biennial Conference of the Association for Machine Translation in the Americas (AMTA), San Diego, CA. find our card
  81. Meyer, Thomas and Bonnie Webber. 2013. Implicitation of discourse connectives in (machine) translation. In Proceedings of the Workshop on Discourse in Machine Translation, pages 19–26, Sofia. find our card
  82. Moschitti, Alessandro. 2006. Efficient convolution kernels for dependency and constituent syntactic trees. In 17th European Conference on Machine Learning, pages 318–329, Berlin. find
  83. Moschitti, Alessandro. 2008. Kernel methods, syntax and semantics for relational text categorization. In Proceedings of the 17th ACM Conference on Information and Knowledge Management, pages 253–262, Napa Valley, CA. find
  84. Nadejde, Maria, Philip Williams, and Philipp Koehn. 2013. Edinburgh’s syntax-based machine translation systems. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 170–176, Sofia. find
  85. Novák, Michal, Anna Nedoluzhko, and Zdeněk Žabokrtský. 2013. Translation of “it” in a deep syntax framework. In Proceedings of the Workshop on Discourse in Machine Translation, pages 51–59, Sofia. find our card
  86. Och, Franz Josef. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of ACL, pages 160–167, Sapporo. find
  87. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the Association for Computational Linguistics, Philadelphia, PA. find our card
  88. Pearson, Karl. 1895. Notes on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, (58):240–242. find
  89. Pitler, Emily and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing Conference, pages 13–16, Suntec. find our card
  90. Popescu-Belis, Andrei, Thomas Meyer, Jeevanthi Liyanapathirana, Bruno Cartoni, and Sandrine Zufferey. 2012. Discourse-level annotation over Europarl for machine translation: Connectives and pronouns. In Proceedings of the Eighth International Conference on Language Resources and Evaluation, pages 2716–2720, Istanbul. find
  91. Popovic, Maja and Hermann Ney. 2007. Word error rates: Decomposition over POS classes and applications for error analysis. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 48–55, Prague. find
  92. Prasad, Rashmi, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn discourse treebank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation, pages 921–950, Marrakech. find our card
  93. Punyakanok, Vasin and Dan Roth. 2001. The use of classifiers in sequential inference. In Advances in Neural Information Processing Systems 14, pages 995–1001, Vancouver. ↗
  94. Quirk, Chris, Arul Menezes, and Colin Cherry. 2005. Dependency treelet translation: Syntactically informed phrasal SMT. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 271–279, Ann Arbor, MI. find our card
  95. Shen, Libin and Aravind K. Joshi. 2003. An SVM-based voting algorithm with application to parse reranking. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 9–16, Edmonton. find
  96. Smola, Alex and S.v.n. Vishwanathan. 2003. Fast kernels for string and tree matching. In Advances in Neural Information Processing Systems 15, pages 585–592, Vancouver. find
  97. Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Biennial Conference of the Association for Machine Translation in the Americas (AMTA), pages 223–231, Cambridge, MA. find our card
  98. Spearman, Charles. 1904. The proof and measurement of association between two things. American Journal of Psychology, 15(1):72–101. ↗
  99. Stanojević, Miloš, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015. Results of the WMT15 metrics shared task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 256–273, Lisbon. find
  100. Steele, David. 2015. Improving the translation of discourse markers for Chinese into English. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 110–117, Denver, CO. find
  101. Sutton, Charles, Andrew McCallum, and Khashayar Rohanimanesh. 2007. Dynamic conditional random fields: Factorized probabilistic models for labeling and segmenting sequence data. Journal of Machine Learning Research (JMLR), 8:693–723. ↗
  102. Taboada, Maite and William C. Mann. 2006. Rhetorical structure theory: Looking back and moving ahead. Discourse Studies, 8(3):423–459. ↗
  103. Tai, Kuo Chung. 1979. The tree-to-tree correction problem. Journal of the ACM, 26(3):422–433. ↗
  104. Taira, Hirotoshi, Katsuhito Sudoh, and Masaaki Nagata. 2012. Zero pronoun resolution can improve the quality of j-e translation. In Proceedings of the Sixth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 111–118, Jeju Island. find our card
  105. Tiedemann, Jörg. 2010a. Context adaptation in statistical machine translation using models with exponentially decaying cache. In Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing, pages 8–15, Uppsala. find
  106. Tiedemann, Jörg. 2010b. To cache or not to cache? Experiments with adaptive models in statistical machine translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 189–194, Uppsala. find
  107. Tu, Mei, Yu Zhou, and Chengqing Zong. 2013. A novel translation framework based on rhetorical structure theory. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 370–374, Sofia. find
  108. Tu, Mei, Yu Zhou, and Chengqing Zong. 2014. Enhancing grammatical cohesion: Generating transitional expressions for SMT. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 850–860, Baltimore, MD. find
  109. Ture, Ferhan, Douglas W. Oard, and Philip Resnik. 2012. Encouraging consistent translation choices. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 417–426, Montréal. find
  110. Vickrey, David, Luke Biewald, Marc Teyssier, and Daphne Koller. 2005. Word-sense disambiguation for machine translation. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 771–778, Vancouver. find
  111. Voigt, Rob and Dan Jurafsky. 2012. Towards a literary machine translation: The role of referential cohesion. In Proceedings of the NAACL-HLT 2012 Workshop on Computational Linguistics for Literature, pages 18–25, Montréal. find
  112. Watanabe, Taro, Jun Suzuki, Hajime Tsukada, and Hideki Isozaki. 2007. Online large-margin training for statistical machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 764–773, Prague. find
  113. Webber, Bonnie. 2004. D-LTAG: Extending Lexicalized TAG to Discourse. Cognitive Science, 28(5):751–779. ↗
  114. Webber, Bonnie, Marine Carpuat, Andrei Popescu-Belis, and Christian Hardmeier, editors. 2015. Proceedings of the Second Workshop on Discourse in Machine Translation. Association for Computational Linguistics, Lisbon. Webber, Bonnie, Andrei Popescu-Belis, Katja Markert, and Jörg Tiedemann, editors. 2013. Proceedings of the Workshop on Discourse in Machine Translation. Association for Computational Linguistics, Sofia. find
  115. Webber, Bonnie, Andrei Popescu-Belis, and Jörg Tiedemann, editors. 2017. Proceedings of the Third Workshop on Discourse in Machine Translation. Association for Computational Linguistics, Copenhagen. find
  116. Wong, Billy T. M. and Chunyu Kit. 2012. Extending machine translation evaluation metrics with lexical cohesion to document level. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1060–1068, Jeju Island. find
  117. Wong, Billy T. M., Cecilia F. K. Pun, Chunyu Kit, and Jonathan J. Webster. 2011. Lexical cohesion for evaluation of machine translation at document level. In Proceedings of the 7th International Conference on Natural Language Processing and Knowledge Engineering, pages 238–242, Tokushima. find
  118. Wu, Dekai and Pascale Fung. 2009. Semantic roles for SMT: A hybrid two-pass model. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 13–16, Boulder, CO. find
  119. Xiong, Deyi, Yang Ding, Min Zhang, and Chew Lim Tan. 2013. Lexical chain based cohesion models for document-level statistical machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1563–1573, Seattle, WA. find
  120. Xiong, Deyi, Min Zhang, and Xing Wang. 2015. Topic-based coherence modeling for statistical machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(3):483–493. ↗
  121. Yeh, Alexander. 2000. More accurate tests for the statistical significance of result differences. In Proceedings of the 18th Conference on Computational Linguistics - Volume 2, pages 947–953, Saarbrücken. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/joty-2017-discourse-structure-machine-translation-evaluation/