Elamur Research⌕
☰
home / papers / 2017 / The University of Edinburgh's Neural MT Systems for WMT17

The University of Edinburgh's Neural MT Systems for WMT17

Summary

This system description covers the University of Edinburgh's entries to two WMT17 shared tasks. The news task had 12 directions, pairing English with Czech, German, Latvian, Russian, Turkish and Chinese. The biomedical task covered English into Czech, German, Polish and Romanian. All systems are attentional encoder-decoder models built with Nematus. They extend the team's WMT16 recipe, which combined BPE subwords, back-translated monolingual data and right-to-left reranking.

New elements this year include:

  • deep recurrent architectures (deep transition and stacked variants)
  • layer normalization and the adam optimizer
  • a refined BPE procedure that suppresses rare or unseen subword units
  • weight tying to shrink models
  • copying target-side monolingual text into the source for EN↔TR

Ablation experiments show gains of 2.2–5 BLEU over this year's baselines. Layer normalization and depth account for most of the improvement over the WMT16 systems. Among constrained news submissions, the systems tied for first place in 11 of 12 directions, and they achieved the top biomedical BLEU everywhere. Notably, the deep models were trained on single GPUs, so strong NMT does not require large hardware.

Key points

  • Smaller vocabularies: filtering rare BPE units (fewer than 50 occurrences) reduced the English vocabulary for EN↔DE from 80581 to 51092 symbols, using 90000 merge operations. Sequence length grew by only 0.2%.
  • Deep model settings: the deep transition models used encoder depth 4 and decoder depth 8, and the stacked models used depth 4 on both sides. Weight tying allowed training on GPUs with 8–12GB of memory.
  • Gains over WMT16: single models improved by 1.5–3 BLEU on the six language pairs entered in both years.
  • Ensembling and reranking: independently trained ensembles consistently beat checkpoint ensembles. Right-to-left reranking now adds only 0–1 BLEU, down from about 1 BLEU in 2016.
  • Copied monolingual data (EN↔TR): mixed with parallel and back-translated data at 1:2:2, it gave small gains or no loss. Trained models are released at data.statmt.org/wmt17_systems/.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

92 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 82 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-4739 · arXiv 1708.00726

@inproceedings{sennrich2017university,
  title = {{The University of Edinburgh's Neural MT Systems for WMT17}},
  author = {Sennrich, Rico and Birch, Alexandra and Currey, Anna and Germann, Ulrich and Haddow, Barry and Heafield, Kenneth and Barone, Antonio Valerio Miceli and Williams, Philip},
  booktitle = {Proceedings of the Second Conference on Machine Translation},
  year = {2017},
  doi = {10.18653/v1/w17-4739},
  eprint = {1708.00726},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w17-4739}
}

References

17 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ba, Lei Jimmy, Ryan Kiros, and Geoffrey E. Hinton. 2016. “Layer Normalization.” CoRR, abs/1607.06450. find our card
  2. Koehn, Philipp, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. “Moses: Open Source Toolkit for Statistical Machine Translation.” Pro… find
  3. Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. “Neural Machine Translation by Jointly Learning to Align and Translate.” Proceedings of the International Conference on Learning Representations (ICLR). find our card
  4. Bojar, Ondřej, Ondřej Dušek, Tom Kocmi, Jindřich Libovický, Michal Novák, Martin Popel, Roman Sudarikov, and Dušan Variš. 2016. “CzEng 1.6: Enlarged Czech-English Parallel Corpus with Processing Tools Dockered.” Text, Speech and Dialogue: 19th International Conference, TSD 2016, Brno, Czech Repub… find
  5. Miceli Barone, Antonio Valerio, Jindřich Helcl, Rico Sennrich, Barry Haddow, and Alexandra Birch. 2017. “Deep Architectures for Neural Machine Translation.” Proceedings of the Second Conference on Machine Translation, Volume 1: Research Papers. Copenhagen, Denmark. find
  6. Moore, Robert C. and William Lewis. 2010. “Intelligent Selection of Language Model Training Data.” Proceedings of the ACL 2010 Conference Short Papers. find our card
  7. Buck, Christian, Kenneth Heafield, and Bas van Ooyen. 2014. “N-gram Counts and Language Models from the Common Crawl.” Proceedings of the Language Resources and Evaluation Conference. Reykjavik, Iceland. find our card
  8. Peter, Jan-Thorsten, Hermann Ney, Ondřej Bojar, Ngoc-Quan Pham, Jan Niehues, Alex Waibel, Franck Burlot, François Yvon, Mārcis Pinnis, Valter Šics, Milos Stanojevic, Philip Williams, Frédéric Blain, and Lucia Specia. 2017. “The QT21 Combined Machine Translation System for English to Latvian.” Proc… find
  9. Burlot, Franck and François Yvon. 2017. “Evaluating the morphological competence of Machine Translation Systems.” Proceedings of the Second Conference on Machine Translation (WMT’17). Copenhagen, Denmark. find
  10. Cho, Kyunghyun, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. “Learning Phrase Representations using RNN Encoder– Decoder for Statistical Machine Translation.” Proceedings of the 2014 Conference on Empirical Meth-Press, Ofir and Lio… find
  11. Sennrich, Rico, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. “Nematus: a Toolkit for Neural Machine Translation.” Proceedings of the Demonstrations at the 15th… find our card
  12. Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016a. “Edinburgh Neural Machine Translation Systems for WMT 16.” Proceedings of the First Conference on Machine Translation, Volume 2: Shared Task Papers, 368–373. Berlin, Germany. find our card
  13. Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016b. “Improving Neural Machine Translation Models with Monolingual Data.” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 86– 96. Berlin, Germany. find
  14. Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016c. “Neural Machine Translation of Rare Words with Subword Units.” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1715–1725. Berlin, Germany. find
  15. Wu, Yonghui, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian,… find
  16. Zeiler, Matthew D. 2012. “ADADELTA: An Adaptive Learning Rate Method.” CoRR, abs/1212.5701. find our card
  17. Zhou, Jie, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. 2016. “Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation.” Transactions of the Association for Computational Linguistics, 4:371–383. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/sennrich-2017-university-edinburgh-s-neural-mt/