Elamur Research
☰
home / papers / 2016 / Exploiting Source-side Monolingual Data in Neural Machine Translation

Exploiting Source-side Monolingual Data in Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D16-1160.pdf

Summary

Most neural machine translation systems learn only from sentence-aligned bitext. Earlier work showed that target-side monolingual text improves the decoder. Zhang and Zong ask whether plentiful source-language text can similarly strengthen the encoder of an attention-based encoder-decoder NMT (Bahdanau et al., 2014), especially when parallel data is limited. They propose two techniques.

  1. Self-learning: a baseline NMT translates related source-side sentences. The resulting synthetic pairs are then mixed with the real bitext for retraining. Decoder updates on synthetic examples can optionally be frozen.
  2. Multi-task learning: two NMT models share one encoder. One translates. The other learns to map source sentences onto versions reordered toward target (English) word order.

On Chinese-to-English experiments, both methods beat a strong attention-based baseline. The multi-task variant performs better but trains less efficiently. The work matters for low-resource settings because it shows that unlabeled source text is a usable resource for NMT. It also shows that relevance of that text matters more than sheer quantity.

Key points

  • The setting is Chinese-to-English translation, compared against an attention-based NMT baseline built on bidirectional RNN encoders (GRU/LSTM).
  • In the sentence reordering model, the source monolingual sentences are rearranged to resemble English order. The task shares the encoder with the translation model.
  • With large bilingual training data, improvements can still exceed 1.0 BLEU. However, they are smaller than in the small-data scenario.
  • Adding monolingual text unrelated to the domain lowered translation quality. Only relevant data helped.
  • Source-side monolingual data proved more beneficial in NMT than in conventional SMT. The paper contrasts its approach with Luong et al. (2015a), who used an autoencoder or skip-thought objective, and with Cheng et al. (2016b), who used a reconstruction model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

131 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 125 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1160

@inproceedings{zhang2016exploiting,
  title = {{Exploiting Source-side Monolingual Data in Neural Machine Translation}},
  author = {Zhang, Jiajun and Zong, Chengqing},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1160},
  url = {https://doi.org/10.18653/v1/d16-1160}
}

References

36 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  2. Yong Cheng, Shiqi Shen, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016a. Agreement-based joint training for bidirectional attention-based neural machine translation. In Proceedings of AAAI 2016. find
  3. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016b. Semi-supervised learning for neural machine translation. In Proceedings of ACL 2016. find
  4. David Chiang. 2007. Hierarchical phrase-based translation. computational linguistics, 33(2):201–228. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statist… find our card
  5. Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. 2016. Incorporating structural alignment biases into an attentional neural translation model. In Proceedings of NAACL 2016. find
  6. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of ACL 2015. find our card
  7. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  8. Shi Feng, Shujie Liu, Mu Li, and Ming Zhou. 2016. Implicit distortion and fertility models for attention-based encoder-decoder nmt model. arXiv preprint arXiv:1601.03317. — [arXiv:1601.03317](https://arxiv.org/abs/1601.03317) find
  9. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. arXiv preprint arXiv:1601.01073. find our card
  10. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term me… find our card
  11. Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In Proceedings of ACL 2015. find our card
  12. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of EMNLP 2013. find
  13. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Proceedings of NIPS 2015. find
  14. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of ACL 2007, pages 177–180. find
  15. Xiaoqing Li, Jiajun Zhang, and Chengqing Zong. 2016. Towards zero unknown word in neural machine translation. In Proceedings of IJCAI 2016. find
  16. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015a. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114. find our card
  17. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015b. Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP 2015. find
  18. Minh-Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2015c. Addressing the rare word problem in neural machine translation. In Proceedings of ACL 2015. find our card
  19. Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016a. A coverage embedding model for neural machine translation. In Proceedings of EMNLP 2016. find
  20. Haitao Mi, Zhiguo Wang, Niyu Ge, and Abe Ittycheriah. 2016b. Supervised attentions for neural machine translation. In Proceedings of EMNLP 2016. find
  21. Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016c. Vocabulary manipulation for large vocabulary neural machine translation. In Proceedings of ACL 2016. find
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL 2002, pages 311–318. find our card
  23. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proceedings of COLING-ACL 2006. find our card
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. find
  25. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of ACL 2016. find our card
  26. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of NIPS 2014. find
  27. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Coverage-based neural machine translation. In Proceedings of ACL 2016. find
  28. Nicola Ueffing, Gholamreza Haffari, and Anoop Sarkar. 2007. Transductive learning for statistical machine translation. In Proceedings of ACL 2007. find
  29. Chao Wang, Michael Collins, and Philipp Koehn. 2007. Chinese syntactic reordering for statistical machine translation. In Proceedings of EMNLP 2007. find
  30. Zhiguo Wang, Haitao Mi, and Abe Ittycheriah. 2016a. Semi-supervised clustering for short text via deep representation learning. In Proceedings of CoNLL 2016. find
  31. Zhiguo Wang, Haitao Mi, and Abe Ittycheriah. 2016b. Sentence similarity learning by lexical decomposition and composition. arXiv preprint arXiv:1602.07019. find
  32. Hua Wu, Haifeng Wang, and Chengqing Zong. 2008. Domain adaptation for statistical machine translation with domain dictionary and monolingual corpora. In Proceedings of COLING 2008, pages 993–1000. find
  33. Jiajun Zhang, Shujie Liu, Mu Li, Ming Zhou, and Chengqing Zong. 2014. Bilingually-constrained phrase embeddings for machine translation. In Proceedings of ACL 2014. find
  34. Jiajun Zhang, Dakun Zhang, and Jie Hao. 2015. Local translation prediction with global sentence representation. In Proceedings of IJCAI 2015. find
  35. Barret Zoph and Keven Knight. 2016. Transfer learning for low-resource neural machine translation. arXiv preprint arXiv:1604.02201v1. find
  36. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Multi-source neural translation. In Proceedings of NAACL 2016. find