Elamur Research
☰
home / papers / 2016 / Modeling Coverage for Neural Machine Translation

Modeling Coverage for Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P16-1008.pdf

Summary

Attention-based neural machine translation (NMT) does not record which source words it has already handled. This leads to two common errors: some source words are rendered repeatedly (over-translation) and others are dropped (under-translation). Phrase-based SMT avoids this with an explicit coverage vector, but standard NMT has no equivalent. Tu et al. propose NMT-COVERAGE, which attaches a coverage vector to each source annotation. The vector is updated after every attention step and fed back into the attention model, so future attention moves away from words that have already been heavily attended. They study two families. The first is a linguistically motivated scalar coverage that accumulates attention weights, optionally normalized by a predicted fertility. The second is a learned neural coverage, with or without gating. On Chinese–English translation with a Groundhog baseline, both families improve translation and alignment quality, and the gains are largest on long sentences. The work matters because it shows that a classic SMT idea can be transplanted into neural attention and trained end-to-end from data.

Key points

  • Coverage model. The general update is C_i,j = g_update(C_{i−1,j}, α_i,j, Φ(h_j), Ψ).
    • The linguistic variant sums past attention weights for each word.
    • The fertility variant divides that sum by a predicted fertility Φ_j = N·σ(U_f h_j), the expected number of target words per source word.
  • Alignment quality (lower scores are better):
    • Groundhog baseline: AER 54.67, SAER 67.00.
    • Linguistic coverage with fertility: AER 52.13, SAER 64.85.
    • NN coverage with gating (d = 10): AER 50.50, SAER 64.25.
  • Long sentences (> 40 words). Groundhog's translations become too short. The coverage models produce longer, more complete output, and the linguistic (with fertility) and NN (gated, d = 10) variants perform similarly on these inputs.
  • Translation quality. The paper reports significant translation improvements, but the exact BLEU figures are in a section missing from the supplied text.
  • Code. The implementation is released at github.com/tuzhaopeng/NMT-Coverage.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

235 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 224 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1008 · arXiv 1601.04811

@inproceedings{tu2016modeling,
  title = {{Modeling Coverage for Neural Machine Translation}},
  author = {Tu, Zhaopeng and Lu, Zhengdong and Liu, Yang and Liu, Xiaohua and Li, Hang},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1008},
  eprint = {1601.04811},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-1008}
}

References

25 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. ICLR 2015. find our card
  2. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. JMLR. find
  3. Peter E. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. find
  4. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation. find
  5. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In ACL 2015. find our card
  6. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In NAACL 2003. find
  7. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: open source toolkit for statistical machine translation. In ACL… find
  8. Percy Liang, Ben Taskar, and Dan Klein. 2006. Alignment by agreement. In NAACL 2006. find
  9. Yang Liu and Maosong Sun. 2015. Contrastive unsupervised word alignment with non-local features. In AAAI 2015. find
  10. Yong Cheng, Shiqi Shen, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Agreement-based Joint Training for Bidirectional Attention-based Neural Machine Translation. In IJCAI 2016. find
  11. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP 2015. find
  12. David Chiang. 2007. phrase-based translation. CL. find
  13. Franz J. Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. Hierarchical find
  14. Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: encoder–decoder approaches. In SSST 2014. find our card
  15. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP 2014. find
  16. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv. find
  17. Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vylomova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. 2016. Incorporating Structural Alignment Biases into an Attentional Neural Translation Model. In NAACL 2016. find
  18. Michael Collins, Philipp Koehn, and Ivona Kučerová. 2005. Clause restructuring for statistical machine translation. In ACL 2005. find
  19. Shi Feng, Shujie Liu, Mu Li, and Ming Zhou. 2016. Implicit distortion and fertility models for attention-based encoder-decoder nmt model. arXiv. find
  20. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In ACL 2003. find
  21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL 2002. find our card
  22. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681. find
  23. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum Risk Training for Neural Machine Translation. In ACL 2016. find our card
  24. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In NIPS 2014. find
  25. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In ICML 2015. find our card