Elamur Research⌕
☰
home / papers / 2017 / Does Neural Machine Translation Benefit from Larger Context?

Does Neural Machine Translation Benefit from Larger Context?

Paper · 2017 · Sebastien Jean, Stanislas Lauly, Orhan Firat et al.

Summary

Conventional neural machine translation handles each sentence in isolation, so it misses discourse cues, such as pronoun antecedents, that appear in neighbouring text. Jean, Lauly, Firat and Cho test whether an attention-based NMT system improves when it can also see nearby source sentences. Their larger-context model (LC-NMT) adds a second bidirectional GRU encoder for the preceding source sentence, with its own attention module. That module is conditioned on the output of the main source attention, so the model picks context words based on the part of the sentence it is currently translating.

Both the baseline and LC-NMT are trained only for translation. The authors evaluate them on BLEU/RIBES and on the WMT'16 cross-lingual pronoun prediction task for En-Fr and En-De, plus IWSLT'15 En-De as a non-lemmatized check. Extra context helps both translation quality and pronoun choice when training data are scarce, but the benefit mostly disappears with the full corpora. A further finding is that plain attention-based NMT matches the leading purpose-built pronoun predictors. The work argues that discourse effects need more targeted evaluation metrics.

Key points

  • Main data are the WMT'16 pronoun-task corpora: 2,441,410 En-Fr and 2,356,313 En-De sentence pairs. Their target sides are tokenized and lemmatized, and the supplied POS tags are ignored.
  • The French label set contains il, ils, elle, elles, ce, cela and on, plus an OTHER category. The German set has er, sie, es, man and OTHER. Performance is measured by macro-averaged recall.
  • To predict pronouns, the trained translation model fills each REPLACE token in the target sentence. When a sentence has several REPLACE tokens, it searches every combination exhaustively and keeps the one with the highest log-probability.
  • Both models use 620-dim word vectors and 1000-dim encoder and decoder GRUs, and the context encoder produces a 2000-dim context vector. Training uses Adadelta and dropout, with early stopping on validation BLEU.
  • Training sets were cut to 5%, 10%, 20%, 40% and 100% of the pronoun-task data to compare small and large regimes. IWSLT'15 En-De has 194,371 training pairs, with IWSLT'12 (1,700 pairs) used for development and IWSLT'14 (1,305 pairs) for testing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 1,105 works

Cite

Cite as Sebastien Jean, Stanislas Lauly, Orhan Firat and Kyunghyun Cho. 2017. Does Neural Machine Translation Benefit from Larger Context?. arXiv:1704.05135.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1704.05135 · arXiv 1704.05135

@misc{jean2017does,
  title = {{Does Neural Machine Translation Benefit from Larger Context?}},
  author = {Sebastien Jean, Stanislas Lauly, Orhan Firat and Kyunghyun Cho},
  year = {2017},
  doi = {10.48550/arxiv.1704.05135},
  eprint = {1704.05135},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1704.05135}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 .
  2. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi,… — Findings of the 2016 Conference on Machine Translation
  3. Ozan Caglayan, Loı̈c Barrault, and Fethi Bougares. 2016. Multimodal attention for neural machine translation. arXiv preprint arXiv:1609.03976 . — [arXiv:1609.03976](https://arxiv.org/abs/1609.03976) find
  4. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gul-5% 10% 20% 40% En-Fr 32.7 35.7 38.2 33.9 36.7 38.6 En-De 19.8 22.1 24.3 20.9 22.7 23.9 (a) BLEU 100% NMT LC-NMT 27.6 28.8 39.9 39.0 NMT LC-NMT 16.3 17.4 25.6 25.1 Table 1: Translation quality in (a) BLEU and (b) RIBES on the cross-lingual pronoun pred… find
  5. BLEU 19.7 20.7 RIBES 77.8 79.0 NMT LC-NMT Table 3: Translation quality on IWSLT (En-De). phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 . — Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
  6. Josep Crego, Jungi Kim, Guillaume Klein, Anabel Rebollo, Kathy Yang, Jean Senellart, Egor Akhanov, Patrice Brunelle, Aurelien Coquard, Yongchao Deng, et al. 2016. Systran’s pure neural machine translation systems. arXiv preprint arXiv:1610.05540 . — [arXiv:1610.05540](https://arxiv.org/abs/1610.05540) find
  7. Raj Dabre, Yevgeniy Puzikov, Fabien Cromieres, and Sadao Kurohashi. 2016. Sepp Hochreiter and Jürgen Schmidhuber. 1997. The kyoto university cross-lingual pronoun translation system. Long short-term memory. Neural computation In Proceedings of the First Conference on Machine 9(8):1735–1780. Transla… find
  8. Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language Michael Denkowski and Alon Lavie. 2014. Meteor pairs. In Proceedings of the 2010 Conference universal: Language specific translation evaluation on Empirical Methods in Natural Language Pro-for any targe… find
  9. Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-sequence attentional neural machine translation. In ACL. find
  10. Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. Learning to parse and translate improves neural machine translation. arXiv preprint arXiv:1702.03525 . 5% 10% 20% 40% En-Fr 84.0 85.0 85.9 84.8 85.6 86.0 En-De 78.9 80.4 81.4 79.5 80.6 81.5
  11. (b) RIBES 100% NMT LC-NMT 82.0 82.4 86.9 86.4 NMT LC-NMT 76.6 77.3 81.7 81.7 find
  12. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In NAACL.
  13. Alex Graves. 2013. Generating sequences with recurrent neural networks. CoRR abs/1308.0850. find
  14. Liane Guillou, Christian Hardmeier, Preslav Nakov, Sara Stymne, Jörg Tiedemann, Yannick Versley, Mauro Cettolo, Bonnie Webber, and Andrei Popescu-Belis. 2016a. Findings of the 2016 wmt shared task on cross-lingual pronoun predic In Proceedings of the First Conference on Machine Translation. Associa… find
  15. Liane Guillou, Christian Hardmeier, Preslav Nakov, Sara Stymne, Jörg Tiedemann, Yannick Versley, Mauro Cettolo, Bonnie Webber, and Andrei Popescu-Belis. 2016b. Findings of the 2016 wmt shared task on cross-lingual pronoun prediction. In Proceedings of the First Conference on Machine Translation (WM… find
  16. Christian Hardmeier, Preslav Nakov, Sara Stymne, Jörg Tiedemann, Yannick Versley, and Mauro Cettolo. 2015. Pronoun-focused mt and cross-lingual pronoun prediction: Findings o In Proceedings of the Second Workshop on Discourse in Machine Translation. Association for Computational Linguistics, Lisbon… find
  17. Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, and Jacob Eisenstein. 2015. Document context language models. arXiv preprint arXiv:1511.03962 . — [arXiv:1511.03962](https://arxiv.org/abs/1511.03962) find
  18. Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A latent variable recurrent neural network for discourse relation language models. arXiv preprint arXiv:1603.01913 . — [arXiv:1603.01913](https://arxiv.org/abs/1603.01913) find
  19. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Juhani Luotolahti, Jenna Kan-2016. Le, Mohammad Norouzi, Wolfgang Macherey, erva, and Filip Ginter. Krikun, Yuan Cao, Qin Gao, Klaus Cross-lingual pronoun prediction with deep recurrent neural Maxim networks. find
  20. Macherey, et al. 2016. Google’s neural ma-In Proceedings of the First Conference on Machine chine translation system: Bridging the gap between Translation. Association for Computational human and machine translation. arXiv preprint Linguistics, Berlin, Germany, pages 596–601. arXiv:1609.08144 . http… find
  21. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. INTER-SPEECH 2:3. find
  22. Maria Nadejde, Siva Reddy, Rico Sennrich, Tomasz Dwojak, Marcin Junczys-Dowmunt, Philipp Koehn, and Alexandra Birch. 2017. Syntax-aware neural machine translation using ccg. arXiv preprint arXiv:1702.01147 . — [arXiv:1702.01147](https://arxiv.org/abs/1702.01147) find
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 311–318.
  24. Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. 2014. Dropout improves recurrent neural networks for handwriting recognition. In Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on. IEEE, pages 285–290. find
  25. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas. pages 223–231.
  26. Lucia Specia, Stella Frank, Khalil Simaan, and Desmond Elliott. 2016. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Translation, Berlin, Germany. Association for Computational Linguistics.
  27. Sara Stymne. 2016. Feature exploration for cross-lingual pronoun prediction. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 609–615. http://www.aclweb.org/anthology/W/W16/W16-2355. ↗
  28. Tian Wang and Kyunghyun Cho. 2016. Larger-context language modelling. In ACL. find
  29. Bonnie Webber, Marine Carpuat, Andrei Popescu-Belis, and Christian Hardmeier, editors. 2015. Proceedings of the Second Workshop on Discourse in Machine Translation. Association for Computational Linguistics, Lisbon, Portugal. http://aclweb.org/anthology/W15-25. ↗
  30. Paul J Werbos. 1990. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE 78(10):1550–1560. find
  31. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 .
  32. Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. arXiv preprint arXiv:1601.00710 .

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jean-2017-neural-machine-translation-benefit-larger/