Elamur Research⌕
☰
home / papers / 2020 / Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation

Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation

Summary

Non-autoregressive translation (NAR) is often presented as a faster alternative to standard autoregressive (AR) Transformers, with only a small loss in quality. Kasai et al. argue that this comparison has been unfair to AR baselines in three ways. First, encoder and decoder layers are usually split evenly. Second, speed is measured only for one sentence at a time. Third, only the NAR models are trained with knowledge distillation. The authors give a complexity analysis showing that decoding cost is dominated by the decoder. They then test AR and iterative NAR models (CMLM, DisCo) with different encoder/decoder depths on seven standard translation directions, using fairseq. An AR model with a deep encoder and a single-layer decoder matches the accuracy of a 6-6 AR model. It reaches one-sentence speeds close to strong NAR systems and is clearly faster when translating in large batches. NAR models, by contrast, lose accuracy when their decoders are made shallow. The paper proposes a fairer evaluation protocol for fast machine translation research.

Key points

  • Speed is reported with two metrics. S1 translates one sentence at a time. Smax uses batches as large as the GPU allows. Both are wall-clock times relative to a 6-6 AR baseline.
  • On WMT14 EN→DE, AR 12-1 scores 28.3 BLEU, the same as AR 6-6. It runs 2.5× faster under S1 and 1.4× faster under Smax.
    • CMLM 6-6 with 10 iterations scores 27.4 BLEU at 1.9× (S1) but only 0.2× (Smax).
    • NAR models stay below 1× under Smax throughout Table 5.
  • Cutting NAR decoders to one layer hurts quality. CMLM 12-1 with 4 iterations drops to 24.7 BLEU on EN→DE, compared with 26.7 for 6-6. The authors' analysis links this to NAR decoders needing depth to learn target word order.
  • Under full parallelization, an AR decoder's time is quadratic in sequence length, while an NAR decoder's is linear. However, the NAR decoder performs T times more total operations, and T usually needs to be at least 4 to be competitive.
  • Distillation also improves AR models. Applying it to both families shows the AR–NAR accuracy gap is larger than earlier papers reported.

Interesting ideas

  • I1. Shift depth from decoder to encoder. Encoder work is parallel over source positions, while the decoder runs step by step. Moving layers to the encoder is therefore a cheap way to speed up any sequential generator without losing quality.
  • I2. Measure speed under more than one usage scenario. Reporting both interactive (S1) and bulk (Smax) speed shows tradeoffs that a single latency number hides. This applies to benchmarking any inference system, such as taggers or parsers on large corpora.
  • I3. Give baselines the same training tricks. If the proposed method uses distillation, the baseline should too. Otherwise the reported gains may come from the training trick rather than the architecture.
  • I4. Separate total operations from parallel time. A method can win on latency yet lose on throughput once the hardware is saturated. A two-part complexity table makes this visible before running any experiments.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Computing & GPUs

Topics: Machine translation

Cited by

102 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 100 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2006.10369 · arXiv 2006.10369

@misc{kasai2020deep,
  title = {{Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation}},
  author = {Kasai, Jungo and Pappas, Nikolaos and Peng, Hao and Cross, James and Smith, Noah A.},
  year = {2020},
  doi = {10.48550/arxiv.2006.10369},
  eprint = {2006.10369},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.10369}
}

References

65 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR, 2015. URL https://arxiv.org/abs/1409.0473. ↗ our card
  2. Antonio Valerio Miceli Barone, Jindřich Helcl, Rico Sennrich, Barry Haddow, and Alexandra Birch. Deep architectures for neural machine translation. In Proc. of WMT, 2017. URL https://arxiv.org/abs/1707.07631. ↗
  3. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proc. of WMT, 2014. URL… find our card
  4. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turc… find our card
  5. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. Findings of the 2017 conference on machine translation (… find our card
  6. William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. KERMIT: Generative insertion-based modeling for sequences, 2019. URL https://arxiv.org/abs/1906.01604. — [arXiv:1906.01604](https://arxiv.org/abs/1906.01604) ↗
  7. William Chan, Mitchell Stern, Jamie Ryan Kiros, and Jakob Uszkoreit. An empirical study of generation order for machine translation. In Proc. of EMNLP, 2020. URL https://arxiv.org/abs/1910.13437. ↗
  8. Ciprian Chelba, Mia Chen, Ankur Bapna, and Noam Shazeer. Faster transformer decoding: N-gram masked self-attention, 2020. URL https://arxiv.org/abs/2001.04589. — [arXiv:2001.04589](https://arxiv.org/abs/2001.04589) ↗
  9. Raj Dabre and Atsushi Fujita. Recurrent stacking of layers for compact neural machine translation models. In Proc. of AAAI, 2019. URL https://arxiv.org/abs/1807.05353. — [arXiv:1807.05353](https://arxiv.org/abs/1807.05353) ↗
  10. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz. Universal transformers. In Proc. of ICLR, 2019. URL https://arxiv.org/abs/1807.03819. ↗
  11. Chris Dyer, Victor Chahuneau, and Noah A. Smith. A simple, fast, and effective reparameterization of ibm model 2. In Proc. of NAACL, 2013. URL https://homes.cs.washington.edu/˜nasmith/papers/dyer+chahuneau+smith.naacl13.pdf. ↗
  12. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence to sequence learning. In Proc. of ICML, 2017. URL https://arxiv.org/abs/1705.03122. ↗ our card
  13. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke S. Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proc. of EMNLP, 2019. URL https://arxiv.org/abs/1904.09324. ↗ our card
  14. Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. Aligned cross entropy for non-autoregressive machine translation. In Proc. of ICML, 2020a. URL https://arxiv.org/abs/2004.01655. — [arXiv:2004.01655](https://arxiv.org/abs/2004.01655) ↗
  15. Marjan Ghazvininejad, Omer Levy, and Luke Zettlemoyer. Semi-autoregressive training improves mask-predict decoding, 2020b. URL https://arxiv.org/abs/2001.08785. — [arXiv:2001.08785](https://arxiv.org/abs/2001.08785) ↗
  16. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non-autoregressive neural machine translation. In Proc. of ICLR, 2018. URL https://arxiv.org/abs/1711.02281. ↗ our card
  17. Jiatao Gu, Qi Liu, and Kyunghyun Cho. Insertion-based decoding with automatically inferred generation order. TACL, 2019a. URL https://arxiv.org/abs/1902.01370. ↗
  18. Jiatao Gu, Changhan Wang, and Jake Zhao. Levenshtein transformer. In Proc. of NeurIPS, 2019b. URL https://arxiv.org/abs/1905.11006. — [arXiv:1905.11006](https://arxiv.org/abs/1905.11006) ↗
  19. Junliang Guo, Linli Xu, and Enhong Chen. Jointly masked sequence-to-sequence model for non-autoregressive neural machine translation. In Proc. of ACL, July 2020. URL https://www.aclweb.org/anthology/2020.acl-main.36. ↗
  20. Mark Harris. Optimizing parallel reduction in CUDA, 2007. URL https://developer.download.nvidia.com/assets/cuda/files/reduction.pdf. ↗
  21. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mengnan Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhan… find our card
  22. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In Proc. of NeurIPS Deep Learning and Representation Learning Workshop, 2015. URL https://arxiv.org/abs/1503.02531. ↗ our card
  23. Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In Proc. of ICLR, 2017. URL https://arxiv.org/abs/1611.01462. ↗
  24. Łukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In Proc. of ICML, 2018. URL https://arxiv.org/abs/1803.03382. — [arXiv:1803.03382](https://arxiv.org/abs/1803.03382) ↗
  25. Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In Proc. of EMNLP, 2013. URL https://www.aclweb.org/anthology/D13-1176. ↗ our card
  26. Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. Non-autoregressive machine translation with disentangled context transformer. In Proc. of ICML, 2020. URL https://arxiv.org/abs/2001.05136. — [arXiv:2001.05136](https://arxiv.org/abs/2001.05136) ↗
  27. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proc. of ICML, 2020. URL https://arxiv.org/abs/2006.16236. ↗
  28. Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proc. of EMNLP, 2016. URL https://arxiv.org/abs/1606.07947. ↗ our card
  29. Young Jin Kim, Marcin Junczys-Dowmunt, Hany Hassan, Alham Fikri Aji, Kenneth Heafield, Roman Grundkiewicz, and Nikolay Bogoychev. From research to production and back: Ludicrously fast neural machine translation. In Proc. of WNGT, 2019. URL https://www.aclweb.org/anthology/D19-5632. ↗
  30. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In Proc. of ICLR, 2020. URL https://arxiv.org/abs/2001.04451. ↗ our card
  31. Jason D. Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proc. of EMNLP, 2018. URL https://arxiv.org/abs/1802.06901. ↗ our card
  32. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proc. of ACL, 2020. URL https://arxiv.org/abs/1910.1346… ↗ our card
  33. Xiaoya Li, Yuxian Meng, Arianna Yuan, Fei Wu, and Jiwei Li. LAVA NAT: A non-autoregressive translation model with look-around decoding and vocabulary attention, 2020. URL https://arxiv.org/abs/2002.03084. — [arXiv:2002.03084](https://arxiv.org/abs/2002.03084) ↗
  34. Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. Hint-based training for non-autoregressive machine translation. In Proc. of EMNLP, 2019. URL https://arxiv.org/abs/1909.06708. ↗
  35. Jindřich Libovický and Jindřich Helcl. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proc. of EMNLP, 2018. URL https://arxiv.org/abs/1811.04719. ↗ our card
  36. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. TACL, 2020. URL https://arxiv.org/abs/2001.08210. ↗
  37. Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard H. Hovy. FlowSeq: Non-autoregressive conditional sequence generation with generative flow. In Proc. of EMNLP, 2019. URL https://arxiv.org/abs/1909.02480. ↗
  38. Elman Mansimov, Alex Wang, and Kyunghyun Cho. A generalized framework of sequence generation with application to undirected sequence models, 2019. URL https://arxiv.org/abs/1905.12790. — [arXiv:1905.12790](https://arxiv.org/abs/1905.12790) ↗
  39. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In Proc. of ICLR, 2018. URL https://arxiv.org/abs/1710.03740. ↗
  40. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proc. of WMT, 2018. URL https://arxiv.org/abs/1806.00187. ↗ our card
  41. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL Demonstrations, 2019. URL https://arxiv.org/abs/1904.01038. ↗ our card
  42. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proc. of ACL, 2002. URL https://www.aclweb.org/anthology/P02-1040. ↗ our card
  43. Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. Random feature attention. In Proc. of ICLR, 2021. URL https://openreview.net/forum?id=QtTKTdVrFBB. ↗
  44. Matt Post. A call for clarity in reporting BLEU scores. In Proc. of WMT, 2018. URL https://www.aclweb.org/anthology/W18-6319. ↗ our card
  45. Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proc. of EACL, 2017. URL https://arxiv.org/abs/1608.05859. ↗ our card
  46. Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. Guiding non-autoregressive neural machine translation decoding with reordering information. In Proc. of AAAI, 2021. URL https://arxiv.org/abs/1911.02215. — [arXiv:1911.02215](https://arxiv.org/abs/1911.02215) ↗
  47. Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non-autoregressive machine translation with latent alignments. In Proc. of EMNLP, 2020. URL https://arxiv.org/abs/2004.07437. ↗
  48. Jean Senellart, Dakun Zhang, Bo Wang, Guillaume Klein, Jean-Pierre Ramatchandirin, Josep Crego, and Alexander Rush. OpenNMT system description for WNMT 2018: 800 words/sec on a single-core CPU. In Proc. of WNG, 2018. URL https://www.aclweb.org/anthology/W18-2715. ↗
  49. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. of ACL, 2016. URL https://www.aclweb.org/anthology/P16-1162. ↗
  50. Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. In Proc. of AAAI, 2020. URL https://arxiv.org/abs/1911.09320. — [arXiv:1911.09320](https://arxiv.org/abs/1911.09320) ↗
  51. Xing Shi and Kevin Knight. Speeding up neural machine translation decoding by shrinking run-time vocabulary. In Proc. of ACL, 2017. URL http://xingshi.me/data/pdf/ACL2017short.pdf. ↗
  52. Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior. In Proc. of AAAI, 2020. URL https://arxiv.org/abs/1908.07181. — [arXiv:1908.07181](https://arxiv.org/abs/1908.07181) ↗
  53. Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. In Proc. of NeurIPS, 2018. URL https://arxiv.org/abs/1811.03115. — [arXiv:1811.03115](https://arxiv.org/abs/1811.03115) ↗
  54. Mitchell Stern, William Chan, Jamie Ryan Kiros, and Jakob Uszkoreit. Insertion transformer: flexible sequence generation via insertion operations. In Proc. of ICML, 2019. URL https://arxiv.org/abs/1902.03249. — [arXiv:1902.03249](https://arxiv.org/abs/1902.03249) ↗
  55. Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng. Fast structured decoding for sequence models. In Proc. of NeurIPS, 2019. URL https://arxiv.org/abs/1910.11555. — [arXiv:1910.11555](https://arxiv.org/abs/1910.11555) ↗
  56. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Proc. of NeurIPS, 2014. URL https://arxiv.org/abs/1409.3215. ↗
  57. Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, and Kevin Gimpel. ENGINE: Energy-based inference networks for non-autoregressive machine translation. In Proc. of ACL, 2020. URL https://arxiv.org/abs/2005.00850. ↗
  58. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NeurIPS, 2017. URL https://arxiv.org/abs/1706.03762. ↗
  59. Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Proc. of ACL, 2019a. URL https://arxiv.org/abs/1906.01787. ↗ our card
  60. Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. Non-autoregressive machine translation with auxiliary regularization. In Proc. of AAAI, 2019b. URL https://arxiv.org/abs/1902.10245. — [arXiv:1902.10245](https://arxiv.org/abs/1902.10245) ↗
  61. Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In Proc. of ICLR, 2019. URL https://arxiv.org/abs/1901.10430. ↗ our card
  62. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  63. Biao Zhang, Deyi Xiong, and Jinsong Su. Accelerating neural transformer via an average attention network. In Proc. of ACL, 2018. URL https://arxiv.org/abs/1805.00631. ↗
  64. Chunting Zhou, Graham Neubig, and Jiatao Gu. Understanding knowledge distillation in non-autoregressive machine translation. In Proc. of ICLR, 2020. URL https://arxiv.org/abs/1911.02727. — [arXiv:1911.02727](https://arxiv.org/abs/1911.02727) ↗
  65. Jiawei Zhou and Phillip Keung. Improving non-autoregressive neural machine translation with monolingual data. In Proc. of ACL, 2020. URL https://arxiv.org/abs/2005.00932. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kasai-2020-deep-encoder-shallow-decoder-reevaluating/