Elamur Research
☰
home / papers / 2017 / Nematus: a Toolkit for Neural Machine Translation

Nematus: a Toolkit for Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/E17-3017.pdf

Summary

Nematus is an open-source toolkit for Neural Machine Translation (NMT) designed for strong translation quality, ease of use, and easy extension. It grew out of the compact dl4mt-tutorial codebase, is written in Python on top of Theano, and follows the attentional encoder–decoder approach of Bahdanau et al. (2015) with several deliberate changes. The decoder starts from the averaged source annotations instead of the final backward encoder state. It uses a new conditional GRU with attention (cGRU_att), in which two GRU transitions surround the attention step. The decoding order is Look, Update, Generate, which simplifies the code. Training uses cross-entropy with several optimizers and also supports minimum risk training toward sentence-level MT metrics. The toolkit adds ensembling, rescoring, and visualisation tools. The paper reports no new benchmark scores. Its value is that the same codebase produced top WMT and IWSLT 2016 submissions and production systems, such as at WIPO, so its design choices matter to both researchers and practitioners.

Key points

  • Architecture changes from Bahdanau et al. (2015) include:
    • a tanh feedforward layer instead of maxout before the softmax;
    • no biases in the embedding layers;
    • optional recurrent Bayesian dropout;
    • factored inputs, with per-feature embeddings concatenated;
    • optional tied embedding matrices.
  • In cGRU_att, recurrence happens only at the level of the whole layer, making it a deep-transition RNN rather than a stacked one.
  • Supported optimizers are SGD, Adadelta, RmsProp, and Adam. Minimum risk training (MRT) can use smoothed sentence BLEU, METEOR, BEER, or mixtures of these, and early stopping is available.
  • Built-in tools cover ensemble decoding (geometric mean of model distributions), beam search, corpus scoring, n-best rescoring, and visualisation of attention weights and beam search graphs.
  • The code is available on GitHub (rsennrich/nematus) under a BSD license.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

189 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 179 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1703.04357

@inproceedings{sennrich2017nematus,
  title = {{Nematus: a Toolkit for Neural Machine Translation}},
  author = {Sennrich, Rico and Firat, Orhan and Cho, Kyunghyun and Birch, Alexandra and Haddow, Barry and Hitschler, Julian and Junczys-Dowmunt, Marcin and Läubli, Samuel and Barone, Antonio Valerio Miceli and Mokry, Jozef and Nădejde, Maria},
  booktitle = {Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2017},
  eprint = {1703.04357},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1703.04357}
}

References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the International Conference on Learning Representations (ICLR). find our card
  2. Boxing Chen and Colin Cherry. 2014. A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 362–367, Baltimore, Maryland, USA. 4 Pre-trained models for 8 translation directions are available at http://sta… find our card
  3. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany. find
  4. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 85–91, Edinburgh, Scotland. find
  5. Milos Stanojevic and Khalil Sima’an. 2014. BEER: BEtter Evaluation as Ranking. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 414–419, Baltimore, Maryland, USA. find
  6. Salah El Hihi and Yoshua Bengio. 1995. Hierarchical Recurrent Neural Networks for Long-Term Dependencies. In Nips, volume 409. find
  7. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, pages 3104–3112, Montreal, Quebec, Canada. find
  8. Yarin Gal. 2015. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. ArXiv e-prints. find our card
  9. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. find
  10. Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints, abs/1605.02688. find our card
  11. Hakan Inan, Khashayar Khosravi, and Richard Socher. 2016. Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling. CoRR, abs/1611.01462. find
  12. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5 - rmsprop. find
  13. WIPO. 2016. WIPO Develops Cutting-Edge Translation Tool For Patent Documents, Oct. http://www.wipo.int/pressroom/en/ articles/2016/article\\\_0014.html. ↗
  14. Marcin Junczys-Dowmunt and Alexandra Birch. 2016. The University of Edinburgh’s systems submission to the MT task at IWSLT. In The International Workshop on Spoken Language Translation (IWSLT), Seattle, USA. find
  15. Matthew D Zeiler. 2012. ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. find our card
  16. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  17. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. 2016. Recurrent highway networks. arXiv preprint arXiv:1607.03474. find
  18. Razvan Pascanu, Çağlar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. 2014. How to Construct Deep Recurrent Neural Networks. In International Conference on Learning Representations 2014 (Conference Track). find
  19. Ofir Press and Lior Wolf. 2017. Using the Output Embedding to Improve Language Models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL), Valencia, Spain. find our card
  20. Jürgen Schmidhuber. 1992. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234–242. find
  21. Rico Sennrich and Barry Haddow. 2016. Linguistic Input Features Improve Neural Machine Translation. In Proceedings of the First Conference on Machine Translation, Volume 1: Research Papers, pages 83– 91, Berlin, Germany. find our card
  22. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh Neural Machine Translation Systems for WMT 16. In Proceedings of the First Conference on Machine Translation, Volume 2: Shared Task Papers, pages 368–373, Berlin, Germany. find our card
  23. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum Risk Training for Neural Machine Translation. In find our card