Elamur Research
☰
home / papers / 2018 / Context-Aware Neural Machine Translation Learns Anaphora Resolution

Context-Aware Neural Machine Translation Learns Anaphora Resolution

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1117.pdf

Summary

Most machine translation systems handle each sentence separately, so they miss cues from surrounding text that could resolve ambiguity and keep output coherent. The authors extend the Transformer with a context-aware encoder for English→Russian subtitle translation. A context sentence is encoded by a stack that shares its lower layers with the source encoder. Only the final source-encoder layer attends to that context, and a learned gate merges the result with ordinary self-attention. Because context can reach the model through this single attention block alone, its influence can be inspected directly.

Trained on OpenSubtitles2018, the model beats both a sentence-level Transformer and a baseline that simply concatenates the context with the source. Analysis of the attention weights shows that much of the benefit comes from pronoun translation. This matters because Russian pronouns, and the verbs and adjectives that agree with them, must match the grammatical gender of the antecedent. Comparisons with CoreNLP output and human annotations indicate the model learns anaphora resolution implicitly, without hand-built coreference features.

Key points

  • Data: OpenSubtitles2018 English–Russian. The authors sampled 2M training instances after cleaning, plus dev and test sets of 10,000 instances each from movies not seen in training. Text was segmented with BPE, using vocabularies of about 32,000 tokens.
  • Architecture: the first N−1 encoder layers are shared between source and context. A <bos> token marks context sentences. A sigmoid gate combines self-attention with context attention in the last layer, and the decoder is the unchanged Transformer decoder.
  • BLEU gains: +0.7 over the context-agnostic Transformer and +0.6 over the concatenation approach of Tiedemann and Scherrer (2017).
  • Human check on 500 test cases with "it", of which 79% had a noun antecedent in context: CoreNLP matched the annotators in 77% of cases, the model's attention in 72%, and a last-noun heuristic in 54%.
  • The error patterns of CoreNLP and the model's attention differ substantially. The authors suggest that improving the attention component with specialized features could further help translation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution · Machine translation

Cited by

167 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 157 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1117 · arXiv 1805.10163

@inproceedings{voita2018context,
  title = {{Context-Aware Neural Machine Translation Learns Anaphora Resolution}},
  author = {Voita, Elena and Serdyukov, Pavel and Sennrich, Rico and Titov, Ivan},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1117},
  eprint = {1805.10163},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p18-1117}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the Third International Conference on Learning Representations (ICLR 2015). San Diego. find our card
  2. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating Discourse Phenomena in Neural Machine Translation. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. New Orleans,… find our card
  3. Marine Carpuat. 2009. One Translation Per Discourse. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions. Association for Computational Linguistics, Boulder, Colorado, pages 19–27. http://www.aclweb.org/anthology/W09-2404. ↗
  4. Zhengxian Gong, Min Zhang, Chew Lim Tan, and Guodong Zhou. 2012. N-gram-based tense models for statistical machine translation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational L… find
  5. Zhengxian Gong, Min Zhang, and Guodong Zhou. 2011. Cache-based Document-level Statistical Machine Translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Edinburgh, Scotland, UK., pages 909–919. http://www.acl… ↗
  6. Christian Hardmeier. 2012. Discourse in statistical machine translation: A survey and a case study. Discours 11. find
  7. Christian Hardmeier and Marcello Federico. 2010. Modelling Pronominal Anaphora in Statistical Machine Translation. In Proceedings of the seventh International Workshop on Spoken Language Translation (IWSLT). pages 283–289. find
  8. Christian Hardmeier, Preslav Nakov, Sara Stymne, Jörg Tiedemann, Yannick Versley, and Mauro Cettolo. 2015. Pronoun-Focused MT and Cross-Lingual Pronoun Prediction: Findings of the 2015 DiscoMT Shared Task on Pronoun Translation. In Proceedings of the Second Workshop on Discourse in Machine Translat… find
  9. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014b. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Com… find our card
  10. Christian Hardmeier, Jörg Tiedemann, and Joakim Nivre. 2013. Latent anaphora resolution for cross-lingual pronoun prediction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, pages 380–391… find
  11. Thomas Meyer, Andrei Popescu-Belis, Najeh Hajlaoui, and Andrea Gesmundo. 2012. Machine Translation of Labeled Discourse Connectives. In Proceedings of the Tenth Conference of the Association for Machine Translation in the Americas (AMTA). http://www.mt-archive.info/AMTA-2012-Meyer.pdf. ↗
  12. Eva Hasler, Phil Blunsom, Philipp Koehn, and Barry Haddow. 2014. Dynamic topic adaptation for phrase-based mt. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Gothenburg, Sweden, pages 328–337.… find
  13. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017. Does Neural Machine Translation Benefit from Larger Context? In arXiv:1704.05135. ArXiv: 1704.05135. find
  14. Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, and Noah A Smith. 2017. Dynamic entity representations in neural language models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, page… find
  15. Ronan Le Nagard and Philipp Koehn. 2010. Aiding pronoun translation with co-reference resolution. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR. Association for Computational Linguistics, Uppsala, Sweden, pages 252–261. http://www.aclweb.org/anthology/… ↗
  16. Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. Opensubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan. find
  17. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014a. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computati… find our card
  18. Lesly Miculicich Werlen and Andrei Popescu-Belis. 2017. Validation of an automatic metric for the accuracy of pronoun translation (apt). In Proceedings of the Third Workshop on Discourse in Machine Translation. Association for Computational Linguistics, Copenhagen, Denmark, pages 17–25. https://doi.… ↗
  19. Ruslan Mitkov. 1999. Introduction: Special issue on anaphora resolution in machine translation and multilingual nlp. Machine Translation 14(3/4):159– 161. http://www.jstor.org/stable/40006919. ↗
  20. Stefan Riezler and John T. Maxwell. 2005. On some pitfalls in automatic evaluation and significance testing for mt. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Association for Computational Linguistics, Ann Arbor, Mi… find
  21. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1… find
  22. Jinsong Su, Hua Wu, Haifeng Wang, Yidong Chen, Xiaodong Shi, Huailin Dong, and Qun Liu. 2012. Translation model adaptation for statistical machine translation with monolingual topic information. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon… find
  23. Jörg Tiedemann. 2010. Context Adaptation in Statistical Machine Translation Using Models with Exponentially Decaying Cache. In Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing. Association for Computational Linguistics, Uppsala, Sweden, pages 8–15. http://www.ac… ↗
  24. Jörg Tiedemann and Yves Scherrer. 2017. Neural Machine Translation with Extended Context. In Proceedings of the Third Workshop on Discourse in Machine Translation. Association for Computational Linguistics, Copenhagen, Denmark, DISCOMT’17, pages 82–92. https://doi.org/10.18653/v1/W17-4811. ↗ our card
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. Los Angeles. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf. ↗
  26. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting Cross-Sentence Context for Neural Machine Translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Denmark, Copenhagen, EMNLP’17, pages 2816–28… find
  27. Sam Wiseman, Alexander M Rush, and Stuart M Shieber. 2016. Learning global features for coreference resolution. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics… find our card