Elamur Research
☰
home / papers / 2019 / When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion

When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1116.pdf

Summary

Sentence-level NMT often produces translations that read well individually but conflict once placed together in a document, and BLEU barely registers such inconsistencies. The authors study this for English–Russian using OpenSubtitles2018. A context-agnostic Transformer translated 2000 consecutive sentence pairs, and annotators judged them in two stages. In 7% of pairs, both sentences were acceptable alone but wrong together. Deixis, ellipsis and lexical cohesion explained roughly 80% of these cases, so the authors built targeted test sets (6000 examples) for these three phenomena. They also adopt a realistic data assumption: plenty of sentence-level parallel data but only a small document-level portion. Their model, CADec, draws on Deliberation Networks. It keeps a sentence-level first-pass model fixed and trains a second-pass, context-aware decoder on the document-level subset to revise the draft translation. CADec matches the baseline's BLEU while clearly improving consistency. The work provides diagnostics that separate systems BLEU cannot tell apart and shows that context handling can be added with little document-level data.

Key points

  • Human study on 2000 OpenSubtitles2018 sentence pairs translated by a Transformer trained on 6m sentence pairs: 82% were good in context, 11% had at least one bad sentence, and 7% were individually fine but inconsistent together.
  • The errors were caused by deixis (37%, mostly the T-V distinction and speaker/addressee gender), ellipsis (29%, mainly wrong morphological forms and wrong verbs in VP-ellipsis), lexical cohesion (14%), ambiguity (9%), anaphora (6%) and other phenomena (5%).
  • CADec gains 12.2 percentage points over the baseline on lexical cohesion, with no change in BLEU. On ellipsis, both CADec and the concat model improve by 19–51 points; concat does better on inflection and CADec on VP-ellipsis.
  • During training, CADec sometimes receives a corrupted reference as its first-pass translation, with probability p. Raising p from 0 to 0.75 lifts lexical cohesion accuracy from 48.7 to 65.0 but lowers deixis from 84.1 to 80.0, while BLEU stays around 32.3–32.45.
  • BLEU is a poor early-stopping signal: it plateaus after about 40k batches while lexical cohesion keeps improving. Code and test sets are released on GitHub (lena-voita/good-translation-wrong-in-context).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cited by

141 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 136 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1116 · arXiv 1905.05979

@inproceedings{voita2019when,
  title = {{When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion}},
  author = {Voita, Elena and Sennrich, Rico and Titov, Ivan},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1116},
  eprint = {1905.05979},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1116}
}

References

39 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ruchit Agrawal, Turchi Marco, and Negri Matteo. 2018. Contextual Handling in Neural Machine Translation: Look Behind, Ahead and on Both Sides. find
  2. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating Discourse Phenomena in Neural Machine Translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper… find our card
  3. Marine Carpuat and Michel Simard. 2012. The trouble with smt consistency. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 442–449, Montréal, Canada. Association for Computational Linguistics. find
  4. Zhengxian Gong, Min Zhang, and Guodong Zhou. 2011. Cache-based document-level statistical machine translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 909–919, Edinburgh, Scotland, UK. Association for Computational Linguistics. find
  5. Liane Guillou. 2013. Analysing lexical consistency in translation. In Proceedings of the Workshop on Discourse in Machine Translation, pages 10–18, Sofia, Bulgaria. Association for Computational Linguistics. find
  6. Liane Guillou and Christian Hardmeier. 2016. Protest: A test suite for evaluating pronouns in machine translation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. European Language Resources Association (ELRA). find
  7. Machine Translation, pages 54–57. Association for Computational Linguistics. find
  8. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representation (ICLR 2015). find
  9. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. find
  10. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brook Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proce… find
  11. Mikhail Korobov. 2015. Morphological analyzer and generator for russian and ukrainian languages. In Analysis of Images, Social Networks and Texts, volume 542 of Communications in Computer and Information Science, pages 320–332. Springer International Publishing. find
  12. Shaohui Kuang, Deyi Xiong, Weihua Luo, and Guodong Zhou. 2018. Modeling coherence for neural machine translation with dynamic and topic caches. In Proceedings of the 27th International Conference on Computational Linguistics, pages 596–606. Association for Computational Linguistics. find
  13. Christian Hardmeier and Marcello Federico. 2010. Modelling Pronominal Anaphora in Statistical Machine Translation. In Proceedings of the seventh International Workshop on Spoken Language Translation (IWSLT), pages 283–289. find
  14. Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791–4796. Association for Computational Linguistics. find our card
  15. Christian Hardmeier, Preslav Nakov, Sara Stymne, Jörg Tiedemann, Yannick Versley, and Mauro Cettolo. 2015. Pronoun-focused mt and cross-lingual pronoun prediction: Findings of the 2015 discomt shared task on pronoun translation. In Proceedings of the Second Workshop on Discourse in Machine Translati… find
  16. Ronan Le Nagard and Philipp Koehn. 2010. Aiding pronoun translation with co-reference resolution. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, pages 252–261, Uppsala, Sweden. Association for Computational Linguistics. find
  17. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017a. Does Neural Machine Translation Benefit from Larger Context? In arXiv:1704.05135. ArXiv: 1704.05135. find
  18. Sébastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017b. Neural machine translation for cross-lingual pronoun prediction. In Proceedings of the Third Workshop on Discourse in Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. Opensubtitles2018: Statistical rescoring of sentence… find
  19. Sameen Maruf and Gholamreza Haffari. 2018. Document context neural machine translation with memory networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1275– 1284, Melbourne, Australia. Association for Computational Lingu… find
  20. Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947–2954, Brussels, Belgium. Association for… find our card
  21. Jane Morris and Graeme Hirst. 1991. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics (Volume 17), pages 21–48. find our card
  22. Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation. In Proceedings of the Third Conference on Machine Translation: Research Papers , pages 61–72, Belgium, Brussels. Associati… find our card
  23. Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In ICML, volume 80 of JMLR Workshop and Conference Proceedings, pages 3953–3962. JMLR.org. find our card
  24. Martin Popel and Ondrej Bojar. 2018. Training Tips for the Transformer Model. pages 43–70. find
  25. Annette Rios and Don Tuggener. 2017. Co-reference resolution of elided subjects and possessive pronouns in spanish-english statistical machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 65… find
  26. Lorenza Russo, Sharid Loáiciga, and Asheesh Gu-lati. 2012. Improving machine translation of null subjects in italian and spanish. In Proceedings of the Student Research Workshop at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 81–89, Avignon, Fra… find
  27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li… find
  28. Jörg Tiedemann. 2010. Context adaptation in statistical machine translation using models with exponentially decaying cache. In Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing, pages 8–15, Uppsala, Sweden. Association for Computational Linguistics. find
  29. Jörg Tiedemann and Yves Scherrer. 2017. Neural Machine Translation with Extended Context. In Proceedings of the Third Workshop on Discourse in Machine Translation, DISCOMT’17, pages 82–92, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, Los Angeles. find
  31. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274, Melbourne, Australia. Associati… find our card
  32. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting Cross-Sentence Context for Neural Machine Translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP’17, pages 2816– 2821, Denmark, Copenhagen. Association for Computational Linguist… find
  33. Longyue Wang, Zhaopeng Tu, Xiaojun Zhang, Hang Li, Andy Way, and Qun Liu. 2016. A novel approach to dropped pronoun translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 983–993, San Diego… find
  34. Billy T. M. Wong and Chunyu Kit. 2012. Extending machine translation evaluation metrics with lexical cohesion to document level. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1060–1068, Jeju Island,… find
  35. Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017. Deliberation networks: Sequence generation beyond one-pass decoding. In NIPS, Los Angeles. find
  36. Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang. 2018. Modeling Coherence for Discourse Neural Machine Translation. In arXiv:1811.05683. ArXiv: 1811.05683. — [arXiv:1811.05683](https://arxiv.org/abs/1811.05683) find
  37. Kazuhide Yamamoto and Eiichiro Sumita. 1998. Feasibility study for ellipsis resolution in dialogues by machine-learning technique. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2. find
  38. Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. find
  39. Improving the transformer translation model with document-level context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 533–542, Brussels, Belgium. Association for Computational Linguistics. find our card