Elamur Research⌕
☰
home / papers / 2017 / Exploiting Cross-Sentence Context for Neural Machine Translation

Exploiting Cross-Sentence Context for Neural Machine Translation

Summary

Standard neural machine translation handles each sentence in isolation. This makes it hard to resolve ambiguous words or to keep terminology consistent across a document. The authors address this by feeding information from preceding source sentences into an attention-based NMT system. A hierarchical RNN builds the context: a sentence-level RNN encodes each of the K previous sentences, and a document-level RNN combines those encodings into one history vector. The vector enters the translation model in two ways. First, it serves as the starting state of the encoder, the decoder, or both. Second, it acts as an extra input to the decoder at every step, optionally controlled by a context gate that decides how much history each target word needs. Experiments used Nematus on about 1M Chinese–English sentence pairs drawn from document-aligned LDC corpora. Every variant beat the baseline, and the initialization and gated-context methods together gained +2.1 BLEU on average. The work shows that inter-sentence context can be added cheaply and helps document-level translation.

Key points

  • Data and evaluation: Training used 1M sentence pairs (25.4M Chinese and 32.4M English words) from LDC corpora that have document boundaries. NIST05 (MT05) was the dev set, and NIST06 (MT06) and NIST08 (MT08) were the test sets. Scores are case-insensitive BLEU.
  • Setup: The system was built on Nematus with the three previous sentences as context (K = 3), 35K-word vocabularies, 600-dimensional embeddings, and a hidden size of 1000.
  • Main results: Average BLEU rose from 30.57 (Nematus) to 32.67 with Init enc+dec plus Gating Auxi. The Moses phrase-based system scored 28.24.
  • Effect of the gate: Adding the context gate to auxiliary context gave roughly +1 BLEU over the ungated version (31.30 → 32.24 average).
  • Manual error analysis: On 15 test documents (~60 sentences), the model fixed 29 of 38 ambiguity errors and 24 of 32 inconsistency errors. It also introduced 15 new errors. The code is released as LC-NMT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

101 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 87 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1301 · arXiv 1704.04347

@inproceedings{wang2017exploiting,
  title = {{Exploiting Cross-Sentence Context for Neural Machine Translation}},
  author = {Wang, Longyue and Tu, Zhaopeng and Way, Andy and Liu, Qun},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1301},
  eprint = {1704.04347},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d17-1301}
}

References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations. San Diego, USA, pages 1–15. find our card
  2. Iacer Calixto, Qun Liu, and Nick Campbell. 2017. Incorporating global visual features into attention-based neural machine translation. arXiv preprint arXiv:1701.06521 . find
  3. Marine Carpuat and Michel Simard. 2012. The trouble with smt consistency. In Proceedings of the 7th Workshop on Statistical Machine Translation. Montreal, Quebec, Canada, pages 442–449. find
  4. Heeyoul Choi, Kyunghyun Cho, and Yoshua Bengio. 2016. Context-dependent word representation for neural machine translation. arXiv preprint arXiv:1607.00578 . — [arXiv:1607.00578](https://arxiv.org/abs/1607.00578) find
  5. Michael Collins, Philipp Koehn, and Ivona Kucerova. 2005. Clause restructuring for statistical machine translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics. Ann Arbor, Michigan, pages 531–540. find our card
  6. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Assocaition for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Beijing, Chin… find our card
  7. Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017. Does neural machine translation benefit from larger context? arXiv preprint arXiv:1704.05135 . find
  8. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, pages 1700–1709. find our card
  9. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… find
  10. Thang Luong, Hieu Pham, and D. Christopher Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal, pages 1412–1421. find
  11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Philadelphia, Pennsylvania, USA, pages 311–318. find our card
  12. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, et al. 2017. Nematus: a toolkit for neural machine translation. arXiv preprint arXiv:1703.04357 . find our card
  13. Rico Sennrich and Barry Haddow. 2016. Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers, Berlin, Germany, chapter Linguistic Input Features Improve Neural Machine Translation, pages 83–91. find our card
  14. Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. Phoenix, Arizona, pages 3776–3783. find our card
  15. Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. 2015. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management. Me… find
  16. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 2014 Neural Information Processing Systems. Montreal, Canada, pages 3104–3112. find
  17. Zhaopeng Tu, Yang Liu, Zhengdong Lu, Xiaohua Liu, and Hang Li. 2017a. Context gates for neural machine translation. Transactions of the Association for Computational Linguistics . find
  18. Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017b. Neural machine translation with reconstruction. In Proceedings of the 31th AAAI Conference on Artificial Intelligence (AAAI-17). San Francisco, California, USA, pages 3097– 3103. find
  19. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th annual meeting of the Association for Computational Linguistics. Berlin, Germany, pages 76–85. find our card
  20. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In Proceedings of the International Conference on Machine Learning, Deep Learning Workshop. pages 1–8. find our card
  21. Tong Xiao, Jingbo Zhu, Shujie Yao, and Hao Zhang. 2011. Document-level consistency verification in machine translation. In Machine Translation Summit. Xiamen, China, volume 13, pages 131–138. find
  22. Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. arXiv preprint arXiv:1601.00710 . find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2017-exploiting-cross-sentence-context-neural/