Elamur Research⌕
☰
home / papers / 2019 / The NiuTrans Machine Translation Systems for WMT19

The NiuTrans Machine Translation Systems for WMT19

Summary

This paper documents the NiuTrans team's (Northeastern University and NiuTrans Co., Ltd.) entries to the WMT19 news translation shared task. They covered 13 directions: 11 supervised (EN↔{ZH, DE, RU, KK, LT} and GU→EN) and the unsupervised DE↔CS sub-track.

All supervised pairs used one shared pipeline:

  • Architecture: deep pre-norm Transformer variants, including Transformer-RPR and Transformer-DLCL, which learns a weighted combination of all earlier layers.
  • Data: strict corpus filtering, plus back-translation by beam search, unrestricted sampling or top-K sampling.
  • Model improvement: iterative ensemble knowledge distillation that uses multiple teachers to keep the student models diverse.
  • Combination: a greedy search for the best ensemble, followed by hypothesis combination and feature-based reranking (e.g., right-to-left models).

The unsupervised systems were NMT models trained on synthetic data from an unsupervised SMT system, with joint or iterative training on top.

Among constrained submissions, the systems placed first on KK↔EN and GU→EN. They were second on RU→EN and DE↔CS, and third on ZH→EN, LT→EN, EN→RU and EN↔DE. The work shows that one general recipe can compete across both high- and low-resource pairs.

Key points

  • Data filtering: sentences over 100 words, HTML content, duplicates, fast-align scores below -6 and length ratios beyond 3:1 were removed.
  • Back-translation choice: sampling-based back-translation worked best for low-resource pairs (Kazakh, Gujarati, Lithuanian). ZH↔EN gained little from it, apart from a small improvement with top-10 sampling.
  • Ensembling and reranking: the greedy ensemble search gave about 1–1.5 BLEU over the best single model on development sets. Hypothesis combination raised the sentence-level oracle by roughly 5 BLEU.
  • GU→EN: about 0.5M parallel pairs were supplemented with HindEnCorp, converted via Hindi–Gujarati alphabet mapping, and with roughly 3M back-translated sentences. newstest2019 BLEU rose from 3.0 (baseline) to 24.9 (DLCL25RPR with distillation, ensemble and reranking).
  • Unsupervised DE↔CS: the systems used about 24.4M monolingual sentences per language from News Crawl 2007–2018 and 60,000 BPE merges. They reached roughly 18–19 BLEU on newstest2019.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

49 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 47 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-5325

@inproceedings{li2019niutrans,
  title = {{The NiuTrans Machine Translation Systems for WMT19}},
  author = {Li, Bei and Li, Yinqiao and Xu, Chen and Lin, Ye and Liu, Jiqiang and Liu, Hui and Wang, Ziyang and Zhang, Yuhao and Xu, Nuo and Wang, Zeyang and Feng, Kai and Chen, Hexuan and Liu, Tengbo and Li, Yanyang and Wang, Qiang and Xiao, Tong and Zhu, Jingbo},
  booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)},
  year = {2019},
  doi = {10.18653/v1/w19-5325},
  url = {https://doi.org/10.18653/v1/w19-5325}
}

References

18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. An effective approach to unsupervised machine translation. arXiv preprint arXiv:1902.01313. find
  2. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv: Computation and Language. find
  3. Ankur Bapna, Mia Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. 2018. Training deeper neural machine translation models with transparent attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3028–3033, Brussels, Belgium. Association for Computation… find
  4. Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. arXiv preprint arXiv:1804.07755. find
  5. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. find our card
  6. Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 427–436. Association for Computational Linguisti… find
  7. Anthony Rousseau. 2013. Xenc: An open-source tool for data selection in natural language processing. The Prague Bulletin of Mathematical Linguistics, (100):73–82. find
  8. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for wmt 16. arXiv preprint arXiv:1606.02891. find our card
  9. Yongchao Deng, Shanbo Cheng, Jun Lu, Kai Song, Jingang Wang, Shenglan Wu, Liang Yao, Guchun Zhang, Haibo Zhang, Pei Zhang, Changfeng Zhu, and Boxing Chen. 2018. Alibaba’s neural machine translation systems for wmt18. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Tas… find
  10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational… find
  11. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016c. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational L… find
  12. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155. find our card
  13. Felix Stahlberg, Adria de Gispert, and Bill Byrne. 2018. The university of cambridge’s machine translation systems for wmt18. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 508–516, Belgium, Brussels. Association for Computational Linguistics. find
  14. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, et al. 2018. Tensor2tensor for neural machine translation. arXiv preprint arXiv:1803.07416. find our card
  15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  16. Qiang Wang, Bei Li, Jiqiang Liu, Bojian Jiang, Zheyang Zhang, Yinqiao Li, Ye Lin, Tong Xiao, and Jingbo Zhu. 2018. The niutrans machine translation system for wmt18. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 532–538, Belgium, Brussels. Associa… find
  17. Qiang Wang, Bei Li, Tong Xiao, and Jingbo Zhu. 2019. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Italy, Florence. Association for Computational Linguistics. find our card
  18. Tong Xiao, Jingbo Zhu, Hao Zhang, and Qiang Li. 2012. Niutrans: An open source toolkit for phrase-based and syntax-based machine translation. In Proceedings of the ACL 2012 System Demonstrations, ACL ’12, pages 19–24, Stroudsburg, PA, USA. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/li-2019-niutrans-machine-translation-systems-wmt19/