Elamur Research⌕
☰
home / papers / 2016 / Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions

Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions

Summary

The paper asks whether neural machine translation can replace phrase-based statistical systems in a large, purely in-domain production setting such as the United Nations. The authors use the fully aligned six-language subcorpus of the United Nations Parallel Corpus v1.0, which covers Arabic, Chinese, English, French, Russian and Spanish. They train attentional encoder-decoder models with Nematus on BPE subwords and compare them, across all 30 translation directions, against a strong Moses configuration modelled on the UN's in-house systems. For the ten English-centred directions they also add a Hiero baseline. NMT equals or outperforms phrase-based SMT in every direction except French→Spanish. The largest gains are for pairs involving Chinese, and Arabic pairs also improve clearly. The second part presents AmuNMT, a C++ beam-search decoder for Nematus models, and examines speed trade-offs: checkpoint ensembling versus averaging, beam size and output-vocabulary selection. The authors find that one consumer GPU already delivers throughput competitive with the deployed Moses setup. This makes NMT a practical option for organizations with large in-house parallel data.

Key points

  • Data: the 6-way UN subcorpus has 86,307 documents, about 11.4M lines and about 335M English tokens. Official dev and test sets of 4,000 sentences each come from 2015 documents. Text was lowercased and tokenized with Moses, Chinese was segmented with Jieba, and BPE used 30,000 units.
  • For directions involving Chinese, NMT beats PB-SMT by roughly 7–9 BLEU. Hiero improves over PB-SMT for zh–en and en–zh but recovers less than half of the NMT gain.
  • Doubling NMT training from 1.2M to 2.4M iterations (about 8 to 16 days on a GTX 1080) added 0.3–1.3 BLEU. No further benefit appeared beyond about 2M iterations.
  • On en–fr, averaging the last 8 checkpoints (51.35 BLEU, 860 words/s) nearly matches a 4-checkpoint ensemble (51.38 BLEU, 200 words/s). A beam of 5 scores only 0.2 BLEU below the best beam size.
  • AmuNMT's vocabulary selection (K = 75, K′ = 75, about 1,250 items per sentence) leaves BLEU unchanged. A single GPU outperforms 16 threads on server-grade Xeon CPUs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

67 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 64 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{junczysdowmunt2016is,
  title = {{Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions}},
  author = {Junczys-Dowmunt, Marcin and Dwojak, Tomasz and Hoang, Hieu},
  booktitle = {Proceedings of the 13th International Conference on Spoken Language Translation},
  year = {2016},
  url = {https://aclanthology.org/2016.iwslt-1.5/}
}

References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Ziemski, M. Junczys-Dowmunt, and B. Pouliquen, “The United Nations Parallel Corpus v1.0,” in LREC 2016. ELRA, 2016. find our card
  2. T. Luong, I. Sutskever, Q. V. Le, O. Vinyals, and W. Zaremba, “Addressing the rare word problem in neural machine translation,” in ACL, 2015. find our card
  3. R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” CoRR, vol. abs/1508.07909, 2015. find
  4. P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst, “Moses: Open source toolkit for statistical machine translation,” in ACL, 2007, pp. 177–180. find
  5. Q. Gao and S. Vogel, “Parallel implementations of word alignment tool,” in Software Engineering, Testing, and Quality Assurance for Natural Language Processing, 2008, pp. 49–57. find
  6. N. Durrani, A. Fraser, H. Schmid, H. Hoang, and P. Koehn, “Can Markov models over minimal translation units help phrase-based SMT?” in ACL, 2013, pp. 399–405. find
  7. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” CoRR, vol. abs/1301.3781, 2013. find
  8. K. Heafield, I. Pouzyrevsky, J. H. Clark, and P. Koehn, “Scalable modified Kneser-Ney language model estimation,” in ACL, 2013, pp. 690– 696. find our card
  9. H. Johnson, J. D. Martin, G. F. Foster, and R. Kuhn, “Improving translation quality by discarding most of the phrasetable,” in EMNLP-CoNLL, 2007, pp. 967–975. find our card
  10. M. Junczys-Dowmunt, “Phrasal Rank-Encoding: Exploiting phrase redundancy and translational relations for phrase table compression,” Prague Bull. Math. Linguistics, vol. 98, pp. 63–74, 2012. find
  11. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” CoRR, vol. abs/1409.0473, 2014. find our card
  12. R. Sennrich, B. Haddow, and A. Birch, “Edinburgh neural machine translation systems for WMT 16,” in WMT, 2016, pp. 371–376. find our card
  13. R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training recurrent neural networks,” in ICML, 2013, pp. 1310–1318. find
  14. M. D. Zeiler, “ADADELTA: An Adaptive Learning Rate Method,” CoRR, vol. abs/1212.5701, 2012. find our card
  15. S. Jean, K. Cho, R. Memisevic, and Y. Bengio, “On using very large target vocabulary for neural machine translation,” in ACL, 2015, pp. 1–10. find our card
  16. H. Mi, Z. Wang, and A. Ittycheriah, “Vocabulary manipulation for neural machine translation,” in ACL, 2016. find
  17. Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corra… find
  18. M. Junczys-Dowmunt, T. Dwojak, and R. Sennrich, “The AMU-UEDIN submission to the WMT16 news translation task: Attention-based NMT models as feature functions in phrase-based SMT,” in WMT, 2016, pp. 319–325. find
  19. B. Pouliquen, C. Elizalde, M. Junczys-Dowmunt, C. Mazenc, and J. Garcı́a-Verdugo, “Large-scale multiple language translation accelerator at the United Nations,” in MT-Summit XIV, 2013, pp. 345–352. find
  20. R. Knowles and P. Koehn, “Neural interactive translation prediction,” in AMTA, 2016. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/dowmunt-2016-is-neural-machine-translation-ready/