Elamur Research⌕
☰
home / papers / 2019 / Simple and Effective Noisy Channel Modeling for Neural Machine Translation

Simple and Effective Noisy Channel Modeling for Neural Machine Translation

Paper · 2019 · Kyra Yee, Yann Dauphin and Michael Auli

Summary

The paper revisits noisy channel modeling for neural machine translation. Bayes' rule splits translation into a reverse channel model p(x|y) and a target language model p(y), and the language model can be trained on unpaired monolingual text. Earlier neural work (Yu et al., 2017) used incremental latent-variable models that read only source prefixes and fell short of ordinary seq2seq quality. The authors instead use a plain Transformer, trained on complete sentence pairs, as the channel model. During decoding it scores the whole source against partial target prefixes, despite the train/test mismatch. Decoding uses a two-stage beam search in which the direct model pre-prunes candidates. The direct, channel and language model scores are then combined linearly with length normalization. Experiments cover four translation directions and use big Transformer LMs trained on WMT newscrawl. In n-best re-ranking, the method beats single direct models, direct ensembles and right-to-left rerankers, with gains of up to 3.2 BLEU on WMT'17 De-En. This shows that a standard seq2seq channel model is a simple and strong way to exploit large monolingual corpora.

Key points

  • Channel model choice: On full targets, a standard seq2seq channel model outperforms a model trained on all target prefixes by 5.7 BLEU. The prefix-trained model is only slightly better for prefixes under about 15 tokens.
  • Language models: Two 12-block big Transformer LMs were trained, one on German newscrawl (260M sentences) and one on English newscrawl (193M sentences). Both use 32K BPE and were trained for 4 days on 32 V100 GPUs.
  • Online decoding (WMT De-En news2017, k1=5, k2=10): CH + DIR + LM gains 1.9 BLEU over DIR and 0.9 BLEU over DIR ENS. Removing per-word normalization hurts accuracy.
  • Re-ranking (news2016): Gains reach up to 4.0 BLEU over DIR, 2.9 over the ensemble and 1.4 over DIR + RL + LM. The channel approach benefits most from larger n-best lists, adding 2.4 BLEU from k1=5 to k1=100.
  • Test sets and code: CH + DIR + LM scores best on all four test sets: WMT De-En 37.7, En-De 30.5, Zh-En 25.6 and IWSLT'14 De-En 35.5. Code and models are released in fairseq.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🏢MetaMenlo Park · United States158 works · 211,459 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksMachine translation · 958 works

Cite

Cite as Kyra Yee, Yann Dauphin and Michael Auli. 2019. Simple and Effective Noisy Channel Modeling for Neural Machine Translation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1571

@inproceedings{yee2019simple,
  title = {{Simple and Effective Noisy Channel Modeling for Neural Machine Translation}},
  author = {Kyra Yee, Yann Dauphin and Michael Auli},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1571},
  url = {https://doi.org/10.18653/v1/d19-1571}
}

References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv, abs/1809.10853. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR.
  3. Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2):263–311. find
  4. Yongchao Deng, Shanbo Cheng, Jun Lu, Kai Song, Jingang Wang, Shenglan Wu, Liang Yao, Guchun Zhang, Haibo Zhang, Pei Zhang, Changfeng Zhu, and Boxing Chen. 2018. Alibaba’s neural machine translation systems for wmt18. In Proc. of WMT. find
  5. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018a. Understanding back-translation at scale. In Proc. of EMNLP. find
  6. Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018b. Classical structured prediction losses for sequence to sequence learning. In Proc. of NAACL.
  7. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. In Proc. of ICML.
  8. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv, abs/1503.03535.
  9. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, Will Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhiru…
  10. Marcin Junczys-Dowmunt. 2018. Microsoft’s submission to the wmt2018 news translation task: How i learned to stop worrying and love the data. In Proc. of WMT. find
  11. Dan Klein and Christopher Manning. 2001. Conditional structure versus conditional estimation in nlp. In Proc. of EMNLP. find
  12. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Proc. of NIPS.
  13. Yuguang Wang, Shanbo Cheng, Liyang Jiang, Jiajun Yang, Wei Chen, Muze Li, Lin Shi, Yanfeng Wang, and Hongtao Yang. 2017. Sogou neural machine translation systems for wmt17. In Proceedings of the Second Conference on Machine Translation, pages 410–415. find
  14. Weijia Xu and Marine Carpuat. 2018. The university of maryland’s chinese-english neural machine translation systems at wmt18. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 535–540. find
  15. Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomás Kociský. 2017. The neural noisy channel. In Proc. of ICLR. find
  16. Philipp Koehn, Kevin Duh, and Brian Thompson. 2018. The jhu machine translation systems for wmt 2018. In Proc. of WMT. find
  17. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proc. of NAACL. find
  18. Gurvan L’Hostis, David Grangier, and Michael Auli. 2016. Vocabulary selection strategies for neural machine translation. arXiv, abs/1610.00072. — [arXiv:1610.00072](https://arxiv.org/abs/1610.00072) find
  19. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair’s wmt19 news translation task submission. In Proc. of WMT.
  20. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL System Demonstrations.
  21. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proc. of WMT.
  22. Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv, abs/1804.08771.
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proc. of ACL. find
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proc. of ACL.
  25. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In Proc. of NIPS. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yee-2019-simple-effective-noisy-channel-modeling/