Elamur Research
☰
home / papers / 2019 / MASS: Masked Sequence to Sequence Pre-training for Language Generation

MASS: Masked Sequence to Sequence Pre-training for Language Generation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1905.02450

Summary

Pre-training methods like BERT work well for language understanding, but they train only an encoder. That makes them a poor fit for generation tasks, which use a full encoder-decoder. MASS addresses this with one pre-training objective that trains both halves together on unlabeled text. A contiguous span of a sentence is replaced with [M] tokens before it goes to the encoder. The decoder then has to reconstruct only that span. On the decoder side, tokens that the encoder could see are hidden, so the decoder has to draw on the encoder's representations. The authors pre-train one Transformer on WMT monolingual corpora covering several languages. They fine-tune it for three tasks across eight datasets: machine translation (unsupervised and low-resource), abstractive summarization and dialogue response generation. MASS beats models with no pre-training and earlier pre-training approaches in both zero-resource and low-resource settings. Its unsupervised English-French result is better than an early supervised attention-based NMT system. This suggests that pre-training the whole sequence-to-sequence model pays off for data-poor generation tasks.

Key points

  • Unsupervised NMT on English-French reaches 37.5 BLEU, a state-of-the-art result. That is more than 4 BLEU above XLM (Lample & Conneau, 2019) on En-Fr and about 1 BLEU above it on Fr-En.
  • Evaluation data: WMT14 English-French, WMT16 English-German and WMT16 English-Romanian for translation, Gigaword for summarization and the Cornell Movie Dialog corpus for response generation.
  • The masked span length k is the key hyperparameter. Setting k = 1 matches BERT-style masked language modeling, and setting k = sentence length matches standard language modeling. Both pre-training perplexity and downstream scores are best when k is about 50% of the sentence, which became the default.
  • Ablation on unsupervised En-Fr: masking scattered tokens instead of a contiguous span ("Discrete") gives 36.9 BLEU. Feeding the decoder all tokens ("Feed") gives 35.3 BLEU. Full MASS gives 37.5.
  • For unsupervised NMT, fine-tuning uses only a back-translation loss on monolingual data, with no extra denoising auto-encoder loss. The code is available at github.com/microsoft/MASS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Machine translation

Cited by

376 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 353 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1905.02450 · arXiv 1905.02450

@misc{song2019mass,
  title = {{MASS: Masked Sequence to Sequence Pre-training for Language Generation}},
  author = {Song, Kaitao and Tan, Xu and Qin, Tao and Lu, Jianfeng and Liu, Tie-Yan},
  year = {2019},
  doi = {10.48550/arxiv.1905.02450},
  eprint = {1905.02450},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1905.02450}
}

References

59 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ando, R. K. and Zhang, T. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(Nov):1817– 1853, 2005. find
  2. Artetxe, M., Labaka, G., Agirre, E., and Cho, K. Unsupervised neural machine translation. CoRR, 2017. find our card
  3. Ayana, Shen, S., Liu, Z., and Sun, M. Neural headline generation with minimum risk training. ArXiv, 2016. find
  4. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. ICLR 2015, 2015a. find our card
  5. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. 2015b. find our card
  6. Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155, 2003. find
  7. Blitzer, J., McDonald, R., and Pereira, F. Domain adaptation with structural correspondence learning. In EMNLP, pp. 120–128. Association for Computational Linguistics, 2006. find
  8. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. In EMNLP, 2015. find
  9. Brown, P. F., Desouza, P. V., Mercer, R. L., Pietra, V. J. D., and Lai, J. C. Class-based n-gram models of natural language. Computational linguistics, 18(4):467–479, 1992. find
  10. Cho, K., van Merrienboer, B., Gülçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, 2014. find
  11. Collobert, R. and Weston, J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML, pp. 160–167. ACM, 2008. find
  12. Dai, A. M. and Le, Q. V. Semi-supervised sequence learning. In NIPS, pp. 3079–3087, 2015. find
  13. Danescu-Niculescu-Mizil, C. and Lee, L. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In ACL Workshop, 2011. find
  14. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, 2018. find
  15. Fedus, W., Goodfellow, I., and Dai, A. Maskgan: Better text generation via filling in the . In ICLR, 2018. find
  16. Firat, O., Sankaran, B., Al-Onaizan, Y., Vural, F. T. Y., and Cho, K. Zero-resource translation with multi-lingual neural machine translation. In EMNLP, pp. 268–277, 2016. find
  17. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N. Convolutional sequence to sequence learning. In ICML, volume 70, pp. 1243–1252, 2017. find our card
  18. Girshick, R., Donahue, J., Darrell, T., and Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, pp. 580–587, 2014. find our card
  19. Graff, David, Kong, Junbo, Chen, Ke, Maeda, and Kazuaki. English gigaword. In Linguistic Data Consortium, 2003. find
  20. Hassan, H., Aue, A., Chen, C., Chowdhary, V., Clark, J., Federmann, C., Huang, X., Junczys-Dowmunt, M., Lewis, W., Li, M., et al. Achieving human parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567, 2018. find our card
  21. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, pp. 770–778, 2016. find
  22. He, T., Tan, X., Xia, Y., He, D., Qin, T., Chen, Z., and Liu, T.-Y. Layer-wise coordination between encoder and decoder for neural machine translation. In Advances in Neural Information Processing Systems, pp. 7944–7954, 2018. find
  23. Howard, J. and Ruder, S. Universal language model fine-tuning for text classification. In ACL, volume 1, pp. 328– 339, 2018. find
  24. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR, 2015. find
  25. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. Skip-thought vectors. In NIPS, pp. 3294–3302, 2015. find
  26. Lample, G. and Conneau, A. Cross-lingual language model pretraining. CoRR, abs/1901.07291, 2019. find
  27. Lample, G., Conneau, A., Denoyer, L., and Ranzato, M. Unsupervised machine translation using monolingual corpora only. CoRR, 2017. find our card
  28. Lample, G., Ott, M., Conneau, A., Denoyer, L., and Ranzato, M. Phrase-based & neural unsupervised machine translation. In EMNLP, pp. 5039–5049, 2018. find
  29. Le, Q. and Mikolov, T. Distributed representations of sentences and documents. In ICML, pp. 1188–1196, 2014. find our card
  30. Leng, Y., Tan, X., Qin, T., Li, X.-Y., and Liu, T.-Y. Unsupervised pivot translation for distant languages. In ACL, 2019. find
  31. Logeswaran, L. and Lee, H. An efficient framework for learning sentence representations. CORR, 2018. find our card
  32. McCann, B., Bradbury, J., Xiong, C., and Socher, R. Learned in translation: Contextualized word vectors. In NIPS, pp. 6294–6305, 2017. find our card
  33. Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., and Khudanpur, S. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association, 2010. find
  34. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. In NIPS, pp. 3111–3119, 2013. find
  35. Ouyang, W., Li, H., Zeng, X., and Wang, X. Learning deep representation with large-scale attributes. In CVPR, pp. 1895–1903, 2015. find
  36. Pennington, J., Socher, R., and Manning, C. Glove: Global vectors for word representation. In EMNLP, pp. 1532– 1543, 2014. find
  37. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In NAACL, volume 1, pp. 2227–2237, 2018. find
  38. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. 2018. find
  39. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. CoRR, 2016. find our card
  40. Ramachandran, P., Liu, P. J., and Le, Q. V. Unsupervised pretraining for sequence to sequence learning. CoRR, abs/1611.02683, 2016. find
  41. Ren, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y. Almost unsupervised text to speech and automatic speech recognition. In ICML, 2019. find
  42. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In ACL, volume 1, pp. 1715–1725, 2016. find
  43. Shang, L., Lu, Z., and Li, H. Neural responding machine for short-text conversation. In ACL, volume 1, pp. 1577– 1586, 2015. find our card
  44. Shen, Y., Tan, X., He, D., Qin, T., and Liu, T.-Y. Dense information flow for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1294–1303, June 201… find
  45. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pp. 1631–1642, 2013. find
  46. Song, K., Tan, X., He, D., Lu, J., Qin, T., and Liu, T.-Y. Double path networks for sequence to sequence learning. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 3064–3074, 2018. find
  47. Suzuki, J. and Nagata, M. Cutting-off redundant repeating generations for neural abstractive summarization. In ACL, pp. 291–297, 2017. find
  48. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In CVPR, pp. 1–9, 2015. find
  49. Tan, X., Ren, Y., He, D., Qin, T., and Liu, T.-Y. Multilingual neural machine translation with knowledge distillation. In ICLR, 2019. find
  50. Tjong Kim Sang, E. F. and De Meulder, F. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In NAACL, pp. 142–147. Association for Computational Linguistics, 2003. find
  51. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS, pp. 6000–6010, 2017. find
  52. Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. Extracting and composing robust features with denoising autoencoders. In ICML, pp. 1096–1103. ACM, 2008. find
  53. Vinyals, O. and Le, Q. V. A neural conversational model. CoRR, abs/1506.05869, 2015. find our card
  54. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1804.07461, 2018. find our card
  55. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnic… find
  56. Yang, Z., Chen, W., Wang, F., and Xu, B. Unsupervised neural machine translation with weight sharing. In ACL, pp. 46–55, 2018. find
  57. Yuan, X., Wang, T., Gulcehre, C., Sordoni, A., Bachman, P., Zhang, S., Subramanian, S., and Trischler, A. Machine comprehension by text-to-text neural question generation. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 15–25, 2017. find
  58. Zhang, J. and Zong, C. Exploiting source-side monolingual data in neural machine translation. In EMNLP, pp. 1535– 1545, 2016. find our card
  59. Zoph, B., Yuret, D., May, J., and Knight, K. Transfer learning for low-resource neural machine translation. In EMNLP, pp. 1568–1575, 2016. find