Elamur Research
☰
home / papers / 2019 / Mask-Predict: Parallel Decoding of Conditional Masked Language Models

Mask-Predict: Parallel Decoding of Conditional Masked Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1633.pdf

Summary

Standard neural machine translation produces output one token at a time, which makes decoding slow. Earlier non-autoregressive (parallel) methods were faster but lost considerable accuracy. The authors propose conditional masked language models (CMLMs). These are transformer encoder-decoders whose decoder attends in both directions. They are trained, in the style of BERT, to recover a randomly sized set of masked target tokens given the source sentence and the remaining target words. A special LENGTH token on the encoder side predicts the length of the output.

For inference, the paper introduces mask-predict. It first generates every target token in parallel, then repeatedly hides and regenerates the tokens with the lowest confidence over a fixed number of cycles.

Experiments on WMT benchmarks for English–German, English–Romanian and English–Chinese show large improvements over earlier parallel decoders. Quality comes close to left-to-right transformers while decoding is considerably faster. This suggests masked language models can serve efficient text generation, not only representation learning.

Key points

  • Data: WMT'14 EN-DE (4.5M pairs), WMT'16 EN-RO (610k) and WMT'17 EN-ZH (20M), each evaluated in both directions. Text is segmented with BPE and scored with BLEU, or SacreBLEU for EN→ZH.
  • Decoding schedule: the number of re-masked tokens shrinks linearly with each iteration. Several length candidates are decoded in parallel, and the output with the best average log-probability is kept.
  • Comparison with parallel decoders: 4 iterations already beat the strongest previous parallel models. At 10 iterations, the method exceeds Lee et al. (2018) by 4–5 BLEU on EN-DE and by up to 3 BLEU on EN-RO.
  • Comparison with autoregressive models: the method gives up at most about 2 BLEU in exchange for roughly a 3x decoding speed-up, and is often within about 1 BLEU.
  • Limitations and resources: CMLMs depend strongly on knowledge distillation from an autoregressive teacher, though the gap is smaller on EN-RO. The code is released at github.com/facebookresearch/Mask-Predict.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

217 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 213 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1633

@inproceedings{ghazvininejad2019mask,
  title = {{Mask-Predict: Parallel Decoding of Conditional Masked Language Models}},
  author = {Ghazvininejad, Marjan and Levy, Omer and Liu, Yinhan and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1633},
  url = {https://doi.org/10.18653/v1/d19-1633}
}

References

19 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  2. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional Sequence to Sequence Learning. ArXiv e-prints. find our card
  3. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369–376. ACM. find
  4. Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In ICLR. find our card
  5. Jiatao Gu, Qi Liu, and Kyunghyun Cho. 2019. Insertion-based decoding with automatically inferred generation order. arXiv preprint arXiv:1902.01370. find
  6. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference for Learning Representations. find
  7. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  8. Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1173– 1182, Brussels, Belgium. Association for Computational Linguis… find our card
  9. Jindřich Libovický and Jindřich Helcl. 2018. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3016– 3021, Brussels, Belgium. Association for Computa… find
  10. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In International Conference on Learning Representations. find
  11. Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. 2019. compare-mt: A tool for holistic comparison of language generation systems. In Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL) Demo Track, Minneapolis, USA. find
  12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  13. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. find our card
  14. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li… find
  15. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450. find our card
  16. Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion transformer: Flexible sequence generation via insertion operations. arXiv preprint arXiv:1902.03249. — [arXiv:1902.03249](https://arxiv.org/abs/1902.03249) find
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  18. Alex Wang and Kyunghyun Cho. 2019. Bert has a mouth, and it must speak: Bert as a markov random field language model. arXiv preprint arXiv:1902.04094. find our card
  19. Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. International Conference on Learning Representations. find our card