Elamur Research
☰
home / papers / 2015 / On Using Monolingual Corpora in Neural Machine Translation

On Using Monolingual Corpora in Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (arXiv)" does not allow redistributing the text itself. Original — arxiv.org/pdf/1503.03535

Summary

Neural machine translation (NMT) relies on large sentence-aligned corpora. Such data is scarce for low-resource language pairs and narrow domains, while target-language monolingual text is plentiful. The authors add a separately pre-trained recurrent language model (RNNLM) to an attention-based NMT system (Bahdanau et al.) in two ways. Shallow fusion reranks beam-search candidates using a weighted sum of the translation-model and LM log-probabilities. Deep fusion concatenates the hidden states of the LM and the NMT decoder, then fine-tunes only the output layer. A learned sigmoid gate scales the LM contribution for each word.

The methods were tested on four tasks:

  • Zh-En SMS/chat and conversational speech (NIST OpenMT'15)
  • Tr-En (IWSLT'14)
  • De-En and Cs-En (WMT'15)

Deep fusion consistently gave the best results. It beat earlier Tr-En systems by up to 1.96 BLEU and gave smaller gains on the high-resource pairs. The size of the gain depended on how closely the monolingual data matched the translation domain. This shows that cheap monolingual text can help NMT, especially when parallel data is limited.

Key points

  • On Tr-En, deep fusion improved on plain NMT by up to +1.19 BLEU and surpassed the strongest earlier published system (Yılmaz et al., 2013) by as much as +1.96 BLEU.
  • On the WMT'15 test sets, deep fusion improved on plain NMT by +0.39 BLEU for De-En (23.61 → 24.00) and +0.47 BLEU for Cs-En (21.89 → 22.36). Shallow fusion gained only 0.09 and 0.29 BLEU. The abstract assigns these two gains to the opposite language pairs, but the table supports the figures given here.
  • Zh-En training used about 430K sentence pairs from DARPA BOLT and GALE data. The abstract reports a gain of up to 1.59 BLEU over phrase-based and hierarchical SMT baselines. On the CTS subset, deep fusion did not help.
  • Domain match drove the gains. LM perplexity on the development sets was 223.68 for Zh-En (news-heavy English Gigaword vs. chat), 163.73 for Tr-En and 78.20 for De-En and Cs-En. The gate was also most active for De-En and Cs-En.
  • The gate bias was initialised at b_g = −1, so the LM is used only when helpful. Separately, copying source words chosen by the attention weights in place of unknown tokens added roughly 1.0 BLEU on De-En and Cs-En.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

150 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 134 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1503.03535 · arXiv 1503.03535

@misc{gulcehre2015using,
  title = {{On Using Monolingual Corpora in Neural Machine Translation}},
  author = {Gulcehre, Caglar and Firat, Orhan and Xu, Kelvin and Cho, Kyunghyun and Barrault, Loic and Lin, Huei-Chi and Bougares, Fethi and Schwenk, Holger and Bengio, Yoshua},
  year = {2015},
  doi = {10.48550/arxiv.1503.03535},
  eprint = {1503.03535},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1503.03535}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  2. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. Wit3: Web inventory of transcribed and translated talks. Proceedings of the 16th Conference of the European Association for Machine Translation (EAMT), pages 261–268. find our card
  3. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 263–270. Association for Computational Linguistics. find our card
  4. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), October.… find
  5. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Association for Computational Linguistics. find
  6. Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. 2013. Maxout networks. In Proceedings of The 30th International Conference on Machine Learning, pages 1319–1327. find
  7. Alex Graves. 2011. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pages 2348– 2356. find
  8. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. find
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  10. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2014. On using very large target vocabulary for neural machine translation. arXiv preprint arXiv:1412.2007. find our card
  11. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1700–1709. Association for Computational Linguistics. find
  12. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs.LG], December. find
  13. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 48–54. Association for Computational Linguis… find
  14. Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York, NY, USA. find
  15. Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2014. Addressing the rare word problem in neural machine translation. arXiv preprint arXiv:1410.8206. find our card
  16. Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and Jan Cernocky. 2011. Rnnlm-recurrent neural network language modeling toolkit. In Proc. of the 2011 ASRU Workshop, pages 196–201. find
  17. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning (ICML 2013). find
  18. R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio. 2014. How to construct deep recurrent neural networks. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014), April. find
  19. Haşim Sak, Tunga Güngör, and Murat Saraçlar. 2007. Morphological disambiguation of turkish text with perceptron algorithm. In Computational Linguistics and Intelligent Text Processing, pages 107–118. Springer. find
  20. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11):2673–2681. find
  21. Holger Schwenk. 2007a. Continuous space language models. Comput. Speech Lang., 21(3):492–518, July. find
  22. Holger Schwenk. 2007b. Continuous space language models. Computer Speech & Language, 21(3):492–518. find
  23. Holger Schwenk. 2012. Continuous space translation models for phrase-based statistical machine translation. In Martin Kay and Christian Boitet, editors, Proceedings of the 24th International Conference on Computational Linguistics (COLIN), pages 1071–1080. Indian Institute of Technology Bombay. find
  24. Nakatani Shuyo. 2010. Language detection library for java. find
  25. Ilya Sutskever, Oriol Vinyals, and Quoc Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS 2014), December. find
  26. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4. find
  27. Ertugrul Yılmaz, Ilknur Durgar El-Kahlout, Burak Aydın, Zisan Sıla Ozil, and Coskun Mermer. 2013. Tubitak turkish-english submissions for iwslt 2013. Proceedings of the 10th International Workshop on Spoken Language Translation (IWSLT), pages 152–159. find
  28. Matthew D Zeiler. 2012. ADADELTA: An adaptive learning rate method. arXiv:1212.5701 [cs.LG]. find our card