Elamur Research⌕
☰
home / papers / 2018 / Document Context Neural Machine Translation with Memory Networks

Document Context Neural Machine Translation with Memory Networks

Summary

Most neural MT systems translate each sentence in isolation. This ignores discourse phenomena such as pronoun reference and consistent word choice, which can depend on context far beyond the neighbouring sentences. Maruf and Haffari treat document translation as structured prediction. Their factor graph links each target sentence to its own source, the rest of the source document, and the other translations. Because the full likelihood is intractable, they train with pseudo-likelihood. At inference, a block coordinate descent procedure starts from sentence-level translations and refines each sentence while the others stay fixed. The architecture adds two external memory networks to an attentional encoder-decoder (Bahdanau et al.):

  • a source memory built by a hierarchical sentence/document bidirectional RNN;
  • a target memory made of the final decoder states of the current translations.

The memory contexts are injected either into the decoder hidden state or into the output layer. On French, German and Estonian into English, the model makes effective use of both source and target context. It statistically significantly beats prior context-aware NMT in BLEU and METEOR, showing that global document context, including target-side context, helps neural MT.

Key points

  • The model was evaluated on three language pairs: Fr→En, De→En and Et→En. Baselines were a sentence-based NMT system (S-NMT) and the context model of Wang et al. (2017).
  • Two integration variants were explored, Memory-to-Context and Memory-to-Output. Each was tested with only the source memory, only the target memory, or both ("dual").
  • To fit within GPU memory, the sentence-level bidirectional RNN of the source memory is pre-trained with a language-modelling objective. The document-level RNN is trained end-to-end.
  • In a manual consistency check on five Et→En test documents (about 70 sentences), the model translated 22 of 32 repeated words consistently. The sentence-based model managed 16.
  • On 20 words or phrases mistranslated by the baseline, the model corrected 85%, while introducing about 10% new errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 76 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1118

@inproceedings{maruf2018document,
  title = {{Document Context Neural Machine Translation with Memory Networks}},
  author = {Maruf, Sameen and Haffari, Gholamreza},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1118},
  url = {https://doi.org/10.18653/v1/p18-1118}
}

References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Parnia Bahar, Tamer Alkhouli, Jan-Thorsten Peter, Christopher Jan-Steffen Brix, and Hermann Ney. 2017. Empirical investigation of optimization algorithms in neural machine translation. In Conference of the European Association for Machine Translation, pages 13–26, Prague, Czech Republic. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations. find our card
  3. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2017. Evaluating discourse phenomena in neural machine translation. In arXiv:1711.00513. find our card
  4. Julian Besag. 1975. Statistical analysis of non-lattice data. Journal of the Royal Statistical Society. Series D (The Statistician), 24(3):179–195. find
  5. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT 3 : Web inventory of transcribed and translated talks. In Proceedings of the 16 th Conference of the European Association for Machine Translation, pages 261–268. find
  6. Kyunghyun Cho, B van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8). find our card
  7. Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (Short Pap… find
  8. Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vy-molova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. 2016. Incorporating structural alignment biases into an attentional neural translation model. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human La… find
  9. Eva Martı́nez Garcia, Carles Creus, Cristina España-Bonet, and Lluı́s Màrquez. 2017. Using word embeddings to enforce document-level lexical consistency in machine translation. The Prague Bulletin of Mathematical Linguistics, 108:85–96. find
  10. Eva Martı́nez Garcia, Cristina España-Bonet, and Lluı́s Màrquez. 2014. Document-level machine translation as a re-translation process. Procesamiento del Lenguaje Natural, 53:103–110. find
  11. Eva Martı́nez Garcia, Cristina España-Bonet, and Lluı́s Màrquez. 2015. Document-level machine translation with word vector models. In Proceedings of the18th Conference of the European Association for Machine Translation, pages 59–66. find
  12. Zhengxian Gong, Min Zhang, and Guodong Zhou. 2011. Cache-based document-level statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 909–919. Association for Computational Linguistics. find
  13. Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, and Pengcheng Yin. 2017. Dynet: The dynamic neural network tool… find our card
  14. Christian Hardmeier and Marcello Federico. 2010. Modelling pronominal anaphora in statistical machine translation. In International Workshop on Spoken Language Translation, pages 283–289. find our card
  15. Christian Hardmeier, Joakim Nivre, and Jörg Tiedemann. 2012. Document-wide decoding for phrase-based statistical machine translation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1179–1190. Associa… find
  16. Christian Hardmeier, Sara Stymne, Jörg Tiedemann, and Joakim Nivre. 2013. Docent: A document-level decoder for phrase-based statistical machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 193–198. Association… find
  17. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735– 1780. find
  18. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017. Does neural machine translation benefit from larger context? In arXiv:1704.05135. find
  19. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Conference Proceedings: the 10th Machine Translation Summit, pages 79–86. AAMT. find
  20. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… find
  21. Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT ’07, pages 228–231, Stroudsburg, PA, USA. Association for Computational Linguistic… find our card
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics. find our card
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54 th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725. find
  24. Jason R. Smith, Herve Saint-Amand, Chris Callison-Burch, Magdalena Plamada, and Adam Lopez. 2013. Dirt cheap web-scale parallel text from the common crawl. In Proceedings of the Conference of the Association for Computational Linguistics. find our card
  25. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems, pages 2440–2448. MIT Press. find
  26. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems, pages 3104–3112. MIT Press. find
  27. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Exploiting cross-sentence context for neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2816–2821. Association for Computational Linguistics. find our card
  28. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. In Proceedings of the International Conference on Learning Representations. find
  29. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1412–1421. Association for Computational Linguistics. find
  30. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/maruf-2018-document-context-neural-machine-translation/