Elamur Research⌕
☰
home / papers / 2021 / Glancing Transformer for Non-Autoregressive Neural Machine Translation

Glancing Transformer for Non-Autoregressive Neural Machine Translation

Summary

Non-autoregressive translation (NAT) models emit all target tokens at once, which makes them fast. However, they assume target words are independent given the source, so they translate noticeably worse than the autoregressive Transformer. Iterative fixes such as Mask-Predict recover quality but need several decoding passes, which reduces the speed advantage. The authors propose the Glancing Language Model (GLM), a training scheme that keeps the vanilla NAT architecture. They apply it in the Glancing Transformer (GLAT). During training, the model first decodes in parallel and compares its output with the reference. It then reveals a number of reference words that grows with the prediction error and learns to predict the remaining ones. Encoder representations, not [MASK] tokens, fill the unrevealed positions. As the model improves, fewer words are revealed, giving an easy-to-hard curriculum that ends in full single-pass generation. On several WMT directions, GLAT beats earlier single-pass NAT systems and comes close to Transformer quality. This shows that fully parallel decoding can be competitive.

Key points

  • GLAT decodes in one pass with an 8×–15× speedup. It trails the autoregressive Transformer by only 0.25–0.9 BLEU, compared with about 7 BLEU for vanilla NAT.
  • GLAT gains roughly 5 BLEU over vanilla NAT and stays within 0.9 BLEU of the Transformer at a 7.9× speedup.
  • It surpasses the multi-pass Mask-Predict on WMT14 DE-EN and WMT16 RO-EN. On WMT14 DE-EN it even beats the Transformer for references shorter than 20 tokens.
  • In ablations on WMT14 (GLAT: 25.21 EN-DE, 29.84 DE-EN BLEU), the glancing sampling strategy adds 5–6 BLEU over uniform sampling. Feeding encoder representations instead of [MASK] adds 0.2–0.3 BLEU.
  • Hamming distance works better than Levenshtein distance for deciding how many words to reveal, by about 0.7/0.9 BLEU on EN-DE/DE-EN without length reranking. Random selection of the revealed words was adopted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

79 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 77 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.155

@inproceedings{qian2021glancing,
  title = {{Glancing Transformer for Non-Autoregressive Neural Machine Translation}},
  author = {Qian, Lihua and Zhou, Hao and Bao, Yu and Wang, Mingxuan and Qiu, Lin and Zhang, Weinan and Yu, Yong and Li, Lei},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.155},
  url = {https://doi.org/10.18653/v1/2021.acl-long.155}
}

References

35 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yu Bao, Shujian Huang, Tong Xiao, Dongqi Wang, Xinyu Dai, and Jiajun Chen. 2021. Non-autoregressive translation by learning target categorical codes. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag… find
  2. Yu Bao, Hao Zhou, Jiangtao Feng, Mingxuan Wang, Shujian Huang, Jiajun Chen, and Lei Li. 2019. Non-autoregressive transformer by position learning. arXiv preprint arXiv:1911.10677. — [arXiv:1911.10677](https://arxiv.org/abs/1911.10677) find
  3. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 1, pages 1171–1179. find
  4. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In ICML, pages 41–48. find our card
  5. Yuntian Deng and Alexander Rush. 2020. Cascaded text generation with markov transformers. Advances in Neural Information Processing Systems, 33. find
  6. Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020a. Aligned cross entropy for non-autoregressive machine translation. In International Conference on Machine Learning, pages 3515–3523. PMLR. find
  7. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In EMNLP-IJCNLP, pages 6114–6123. find our card
  8. Marjan Ghazvininejad, Omer Levy, and Luke Zettlemoyer. 2020b. Semi-autoregressive training improves mask-predict decoding. arXiv preprint arXiv:2001.08785. — [arXiv:2001.08785](https://arxiv.org/abs/2001.08785) find
  9. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, pages 369–376. find
  10. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In ICLR. find our card
  11. Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Levenshtein transformer. In NeurIPS, pages 11179– 11189. find
  12. Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. 2019. Non-autoregressive neural machine translation with enhanced decoder input. In AAAI, volume 33, pages 3723–3730. find
  13. Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2020a. Fine-tuning by curriculum learning for non-autoregressive neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7839–7846. find
  14. Junliang Guo, Linli Xu, and Enhong Chen. 2020b. Jointly masked sequence-to-sequence model for non-autoregressive neural machine translation. In ACL, pages 376–385. find
  15. Richard W Hamming. 1950. Error detecting and error correcting codes. The Bell system technical journal, 29(2):147–160. find
  16. Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. 2020. Non-autoregressive machine translation with disentangled context transformer. In International Conference on Machine Learning, pages 5144–5155. PMLR. find
  17. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  18. Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In EMNLP, pages 1173–1182. find our card
  19. Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, pages 707–710. find
  20. Xiaoya Li, Yuxian Meng, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. Lava nat: A non-autoregressive translation model with look-around decoding and vocabulary attention. arXiv preprint arXiv:2002.03084. — [arXiv:2002.03084](https://arxiv.org/abs/2002.03084) find
  21. Zhuohan Li, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Hint-based training for non-autoregressive translation. In EMNLP-IJCNLP. find
  22. Jindřich Libovickỳ and Jindřich Helcl. 2018. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In EMNLP, pages 3016–3021. find our card
  23. Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. 2019. FlowSeq: Non-autoregressive conditional sequence generation with generative flow. In EMNLP-IJCNLP, pages 4273– 4283, Hong Kong, China. find
  24. Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. 2019. CGMH: Constrained sentence generation by Metropolis-Hastings sampling. In AAAI. find
  25. Tsvetomila Mihaylova and André FT Martins. 2019. Scheduled sampling for transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 351–356. find
  26. Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. 2019. Guiding non-autoregressive neural machine translation decoding with reordering information. arXiv preprint arXiv:1911.02215. — [arXiv:1911.02215](https://arxiv.org/abs/1911.02215) find
  27. Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. Non-autoregressive machine translation with latent alignments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1098–1108. find
  28. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL, pages 1715–1725. find
  29. Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. 2020. Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. In AAAI, pages 198–205. find
  30. Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. 2020. Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8846–8853. find
  31. Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng. 2019. Fast structured decoding for sequence models. In NeurIPS, pages 3016–3026. find
  32. Zhiqing Sun and Yiming Yang. 2020. An em approach to non-autoregressive conditional sequence generation. In International Conference on Machine Learning, pages 9249–9258. PMLR. find
  33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998–6008. find
  34. Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019. Non-autoregressive machine translation with auxiliary regularization. In AAAI. find
  35. Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019. Imitation learning for non-autoregressive neural machine translation. In ACL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qian-2021-glancing-transformer-non-autoregressive-neural/