Elamur Research
☰
home / papers / 2017 / OpenNMT: Open-Source Toolkit for Neural Machine Translation

OpenNMT: Open-Source Toolkit for Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-4012.pdf

Summary

By 2017, neural machine translation had overtaken rule-based and statistical systems. Most available implementations, however, were either closed industry systems or research code that gave production users little support. The authors, from SYSTRAN and Harvard NLP, released OpenNMT, an MIT-licensed NMT framework that replaces Harvard's earlier seq2seq-attn. It is written in Lua/Torch, and a PyTorch port exposes the same API. It includes attention-based encoder-decoder models, input feeding, stacked gated RNNs, beam search, multi-GPU training, a reversible tokenizer with BPE, tools for pretrained embeddings, and several decoders, including a C engine built on Eigen for running on CPUs. The design has three goals: fast training and inference, modular and readable code, and easy extension to new attention types, factored outputs, or non-text inputs such as images and speech. On WMT15 English→German, OpenNMT trains and translates faster than Nematus and reaches higher BLEU. A single multilingual model covering five Romance languages beats models trained separately for each pair. OpenNMT gives NMT a shared, compact baseline, much as Moses did for SMT.

Key points

  • Code size: the Lua version is about 4K lines and the Python version is under 1K, compared with more than 100K for Moses. Sharing memory buffers between network clones cuts GPU memory use by about 70%.
  • Multi-GPU training: with 8 GPUs, each epoch runs about 6× faster, and reaching a similar loss takes about 3.5× less total time.
  • EN→DE (WMT15, tested on newstest2014) against Nematus: with a 32k BPE vocabulary, OpenNMT gets 19.34 BLEU vs 18.25, and trains at 5254 tok/s vs 3221. With a 50k word vocabulary, it gets 17.60 vs 17.28.
  • Multilingual model: one model covers ES, FR, IT, PT and RO in both directions (20 pairs). It was trained on 4M sentences from OPUS (Europarl, GlobalVoices, Ted) with a shared 32k BPE vocabulary, and gains +3.3 to +10.4 BLEU over models trained on a single pair.
  • Other uses: the im2text extension for mathematical OCR adds under 500 lines of code. OpenNMT also reproduced Gigaword summarization (ROUGE-1 33.13) and trained a chatbot on 14M OpenSubtitles sentences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

711 papers of our library cite it (of the 140673 whose bibliographies we read); 34 of them have a page here, 677 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{klein2017opennmt,
  title = {{OpenNMT: Open-Source Toolkit for Neural Machine Translation}},
  author = {Klein, Guillaume and Kim, Yoon and Deng, Yuntian and Senellart, Jean and Rush, Alexander},
  booktitle = {Proceedings of ACL 2017, System Demonstrations},
  year = {2017},
  url = {https://aclanthology.org/P17-4012/}
}

References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In Proc of EMNLP. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation By Jointly Learning To Align and Translate. In ICLR. pages 1–15. https://doi.org/10.1146/annurev.neuro.26.041002.131047. ↗ our card
  3. André FT Martins and Ramón Fernandez Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. arXiv preprint arXiv:1602.02068 . — [arXiv:1602.02068](https://arxiv.org/abs/1602.02068) find
  4. Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016. pages 10–21.… find our card
  5. G. Neubig. 2017. Neural Machine Translation and Sequence-to-sequence Models: A Tutorial. ArXiv e-prints . find
  6. Graham Neubig. 2013. Travatar: A forest-to-string machine translation engine based on tree transducers. In Proc ACL. Sofia, Bulgaria. find
  7. William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. 2015. Listen, attend and spell. CoRR abs/1508.01211. http://arxiv.org/abs/1508.01211. — [arXiv:1508.01211](https://arxiv.org/abs/1508.01211) ↗
  8. Rico Sennrich and Barry Haddow. 2016. Linguistic input features improve neural machine translation. arXiv preprint arXiv:1606.02892 . find our card
  9. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proc of EMNLP. find
  10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. CoRR abs/1508.07909. http://arxiv.org/abs/1508.07909. ↗
  11. Sumit Chopra, Michael Auli, Alexander M Rush, and SEAS Harvard. 2016. Abstractive sentence summarization with attentive recurrent neural networks. Proceedings of NAACL-HLT16 pages 93–98. find our card
  12. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In NIPS. page 9. http://arxiv.org/abs/1409.3215. ↗
  13. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869 . find our card
  14. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 . find
  15. Jason Weston, Sumit Chopra, 2014. Memory networks. http://arxiv.org/abs/1410.3916. ↗
  16. Josep Crego, Jungi Kim, and Jean Senellart. 2016. Systran’s pure neural machine translation system. arXiv preprint arXiv:1602.06023 . and Antoine Bordes. CoRR abs/1410.3916. find
  17. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 . find
  18. Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. 2012. Large scale distributed deep networks. In Advances in neural information processing systems. pages 1223–1231. find
  19. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. CoRR abs/1502.03044. http://arxiv.org/abs/1502.03044. ↗ our card
  20. Yuntian Deng, Anssi Kanervisto, and Alexander M. Rush. 2016. What you get is what you see: A visual markup decompiler. CoRR abs/1609.04938. http://arxiv.org/abs/1609.04938. ↗
  21. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proc ACL. find our card
  22. Chris Dyer, Jonathan Weese, Hendra Setiawan, Adam Lopez, Ferhan Ture, Vladimir Eidelman, Juri Ganitkevitch, Phil Blunsom, and Philip Resnik. 2010. cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models. In Proc ACL. Association for Computational Lingu… find
  23. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780. find
  24. Mike Schuster Quoc V. Le Maxim Krikun Yonghui Wu Zhifeng Chen Nikhil Thorat Fernanda Vigas Martin Wattenberg Greg Corrado Macduff Hughes Jeffrey Dean Johnson. 2016. Google’s multilingual neural machine translation system: Enabling zero-shot translation . find
  25. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proc ACL. Association for Computational Linguistics, pages 177… find