Elamur Research⌕
☰
home / papers / 2016 / Incorporating Discrete Translation Lexicons into Neural Machine Translation

Incorporating Discrete Translation Lexicons into Neural Machine Translation

Summary

Attentional NMT systems often get rare content words wrong. They substitute a word that fits the context but changes the meaning, for example outputting "Norway" when the source says "Tunisia". Phrase-based systems rarely make this error because they rely on explicit word mappings. Arthur, Neubig and Nakamura add discrete probabilistic translation lexicons to an attentional encoder-decoder in the style of Luong et al. (2015a).

For each source sentence, they build a matrix of word-level lexical probabilities. At every decoding step, the current attention vector weights its columns, producing a lexicon-derived distribution over the next target word. This distribution is merged with the NMT output in one of two ways:

  • as a log-space bias added before the softmax, or
  • through linear interpolation with a learnable coefficient.

The lexicons come from IBM-model alignments of the training data, a handmade dictionary, or a hybrid of both. On two English-Japanese corpora, the approach raises BLEU by 2.0–2.3 and NIST by 0.13–0.44, and training converges faster. The work shows a cheap way to bring the reliability of symbolic resources into neural translation.

Key points

  • Combination methods: The bias variant adds log(p_l + ϵ) to the softmax input, with ϵ = 0.001. The linear variant uses λ = sigmoid(x), with x learned and initialized so that λ = 0.5.
  • Lexicon sources and results: Three lexicon types were tested: auto (alignment-based), man (handmade) and hyb (hybrid). The bias method helped auto and hyb but not man. The linear method helped only man, whose source-vocabulary coverage was 35.3% on BTEC and 9.7% on KFTT. Linear interpolation was generally weaker than the bias method.
  • Attention sharpness: The auto-bias model's attention was more focused than the baseline's, with per-word entropy of 3.23 bits versus 3.81 bits for the attn baseline.
  • Scaling test: On the larger ASPEC corpus (2M sentence pairs, 63M tokens, 81K vocabulary), BLEU rose from 20.82 (attn) to 22.66 (auto-bias).
  • Relation to prior work: Copy mechanisms and the vocabulary restriction of Mi et al. (2016) are framed as special cases of this approach. The authors list context-dependent λ as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

95 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 88 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1162

@inproceedings{arthur2016incorporating,
  title = {{Incorporating Discrete Translation Lexicons into Neural Machine Translation}},
  author = {Arthur, Philip and Neubig, Graham and Nakamura, Satoshi},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1162},
  url = {https://doi.org/10.18653/v1/d16-1162}
}

References

32 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for extreme summarization of source code. In Proceedings of the 33th International Conference on Machine Learning (ICML). find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 4th International Conference on Learning Representations (ICLR). Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neura… find our card
  3. Arianna Bisazza, Nick Ruiz, and Marcello Federico. 2011. Fill-up versus interpolation methods for phrase-based SMT adaptation. In Proceedings of the 2011 International Workshop on Spoken Language Translation (IWSLT), pages 136–143. find our card
  4. Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, pages 263–311. find
  5. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, pages 201–228. find our card
  6. Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–decoder approaches. In Proceedings of the Workshop on Syntax and Structure in Statistical Translation (SSST), pages 103–111. find our card
  7. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1693–1703. Marta R. Costa-Jussà and José A. R. Fon… find our card
  8. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, pages 138–145. find
  9. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648. Felix A. Gers, Jü… find
  10. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1631–1640. Çaglar Gülçehre, Sungjin Ahn, Ramesh Nallapati, Bowen Z… find our card
  11. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1700–1709. find our card
  12. Gen-ichiro Kikui, Eiichiro Sumita, Toshiyuki Takezawa, and Seiichi Yamamoto. 2003. Creating corpora for speech-to-speech translation. In 8th European Conference on Speech Communication and Technology, EU-ROSPEECH 2003 - INTERSPEECH 2003, Geneva, Switzerland, September 1-4, 2003, pages 381–384. find
  13. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. find
  14. CoRR. Dietrich Klakow. 1998. Log-linear interpolation of language models. In Proceedings of the 5th International Conference on Speech and Language Processing (IC-SLP). find
  15. Phillip Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL), pages 48– 54. Philipp Koehn, Hieu Hoang, Alexandra… find
  16. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  17. Percy Liang, Ben Taskar, and Dan Klein. 2006. Alignment by agreement. In Proceedings of the 2006 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL), pages 104–111. find our card
  18. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W. Black. 2015. Character-based neural machine translation. find our card
  19. CoRR. Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1054–1063. find our card
  20. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1412–1421. find
  21. Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the rare word problem in neural machine translation. In Proceedings of the 53th Annual Meeting of the Association for Computational Linguistics (ACL) and the 7th Internationali Joint Conference on Na… find our card
  22. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: The Penn treebank. Computational Linguistics, pages 313–330. find our card
  23. Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016. Vocabulary manipulation for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 124–129. find
  24. Toshiaki Nakazawa, Manabu Yaguchi, Kiyotaka Uchimoto, Masao Utiyama, Eiichiro Sumita, Sadao Kurohashi, and Hitoshi Isahara. 2016. Aspec: Asian scientific paper excerpt corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2016), pages 2204–2208. find our card
  25. Graham Neubig, Yosuke Nakata, and Shinsuke Mori. 2011. Pointwise prediction for robust, adaptable Japanese morphological analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL), pages 529–533. find our card
  26. Graham Neubig. 2011. The Kyoto free translation task. http://www.phontron.com/kftt. ↗
  27. Graham Neubig. 2013. Travatar: A forest-to-string machine translation engine based on tree transducers. In Proceedings of the 51th Annual Meeting of the Association for Computational Linguistics (ACL), pages 91– 96. find
  28. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, pages 19–51. find
  29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318. find our card
  30. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 86–96. find
  31. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, pages 1929–1958. find
  32. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 28th Annual Conference on Neural Information Processing Systems (NIPS), pages 3104– 3112. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/arthur-2016-incorporating-discrete-translation-lexicons-into/