Elamur Research⌕
☰
home / papers / 2020 / ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training

ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training

Summary

ProphetNet addresses a weakness of standard autoregressive Seq2Seq training. Teacher forcing only rewards guessing the immediately next token, so models tend to overfit short-range patterns such as bigrams and neglect longer-range coherence. The authors replace this one-step objective with future n-gram prediction: at each decoder position the model is trained to predict the next n tokens at once. To make this efficient, they extend XLNet's two-stream attention into an n-stream self-attention decoder. A main stream behaves like an ordinary Transformer decoder, and n extra predicting streams each forecast a token further ahead. Because the predicting streams share parameters with the main stream, they can be switched off at inference, which leaves a normal next-token model. Pre-training combines this objective with mask-based denoising auto-encoding on a 16GB corpus and a 160GB corpus. Once fine-tuned, ProphetNet achieves state-of-the-art performance, relative to models pre-trained on similar amounts of data, on two summarization benchmarks (CNN/DailyMail and Gigaword) and on question generation from SQuAD 1.1. This shows that adding a look-ahead training signal can improve generation quality without changing how decoding works.

Key points

  • Training objective: a weighted sum of the usual next-token loss and n−1 extra losses for tokens further ahead. The weights follow a power attenuation scheme controlled by a coefficient γ.
  • Pre-training data:
    • a base-scale set (16GB, the BERT data, Wikipedia+BookCorpus);
    • a large-scale set (160GB, comparable to BART's).
  • Large-scale efficiency: with the 160GB data, ProphetNet sets new state-of-the-art results on CNN/DailyMail and Gigaword. It uses roughly a third of BART's pre-training epochs and about a fifth of the corpus used by T5 and PEGASUS.
  • Without pre-training (CNN/DailyMail dev): ProphetNet reaches ROUGE 40.66/18.05/37.79 (R-1/R-2/R-L), compared with 39.19/17.60/36.69 for a same-sized Transformer.
  • n-gram ablation (base models, CNN/DailyMail test):
    • 2-gram and 3-gram variants score similarly (R-1 42.52 vs 42.61).
    • Both beat MASS base and the 1-gram variant.
    • The 2-gram version was chosen as the default because it trains about 15% faster.

Interesting ideas

  • I1. Auxiliary look-ahead losses as regularization: asking hidden states to support predictions several steps ahead pushes them to encode planning information, not just local continuations. This trick could transfer to any autoregressive model, for example a Ukrainian LM or a tagger that decodes sequentially.
  • I2. Training-only heads that vanish at inference: the extra streams share weights with the main stream and are simply dropped after training. You get a richer training signal while the model keeps standard decoding cost and stays compatible with existing tooling.
  • I3. Decaying weights for distant targets: down-weighting predictions further in the future, with a tunable attenuation γ, makes it easy to balance the auxiliary signal against the main objective. This is a reusable recipe for multi-horizon losses in general.
  • I4. Checking gains without pre-training: the authors also test the objective with no pre-training at all. This separates the effect of the objective from the effect of the pre-training data, an evaluation habit worth copying when proposing new losses.
  • I5. Choosing n by cost as well as score: the 2-gram setting was selected over a marginally better 3-gram one for speed. This is a practical example of reporting the cost-benefit trade-off explicitly rather than chasing the top number.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

77 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 75 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.217

@inproceedings{qi2020prophetnet,
  title = {{ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training}},
  author = {Qi, Weizhen and Yan, Yu and Gong, Yeyun and Liu, Dayiheng and Duan, Nan and Chen, Jiusheng and Zhang, Ruofei and Zhou, Ming},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.217},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.217}
}

References

41 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018. Retrieve, rerank and rewrite: Soft template based neural summarization. In ACL. find
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  3. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In NeurIPS. find our card
  4. Xinya Du and Claire Cardie. 2018. Harvesting paragraph-level question-answer pairs from wikipedia. In ACL. find
  5. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. arXiv preprint arXiv:1705.00106. find our card
  6. Sergey Edunov, Alexei Baevski, and Michael Auli. 2019. Pre-trained language model representations for language generation. arXiv preprint arXiv:1903.09722. find
  7. Angela Fan, David Grangier, and Michael Auli. 2017. Controllable abstractive summarization. arXiv preprint arXiv:1711.05217. find our card
  8. Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. In EMNLP. find our card
  9. Caglar Gulcehre, Francis Dutil, Adam Trischler, and Yoshua Bengio. 2017. Plan, attend, generate: Planning for sequence-to-sequence models. In NIPS. find
  10. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529. find our card
  11. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. find
  12. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In ACL. find our card
  13. David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, and Chris Pal. 2016. Zoneout: Regularizing rnns by randomly preserving hidden activations. arXiv preprint arXiv:1606.01305. — [arXiv:1606.01305](https://arxiv.org/abs/1606.01305) find
  14. Carolin Lawrence, Bhushan Kotnis, and Mathias Niepert. 2019. Attending to future tokens for bidirectional sequence generation. arXiv preprint arXiv:1908.05915. find
  15. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. find our card
  16. Jiwei Li, Will Monroe, and Dan Jurafsky. 2017. Learning to decode for future success. arXiv preprint arXiv:1701.06549. — [arXiv:1701.06549](https://arxiv.org/abs/1701.06549) find
  17. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. find
  18. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345. find
  19. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017. Regularizing and optimizing lstm language models. arXiv preprint arXiv:1708.02182. find our card
  20. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI. find our card
  21. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023. find
  22. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. find our card
  23. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In ICML. find
  24. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. find
  25. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf. ↗
  26. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP. find our card
  29. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685. find
  30. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL. find
  31. Dmitriy Serdyuk, Nan Rosemary Ke, Alessandro Sordoni, Adam Trischler, Chris Pal, and Yoshua Bengio. 2018. Twin networks: Matching the future for sequence generation. In ICLR. find
  32. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450. find our card
  33. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. find
  35. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. find our card
  36. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. arXiv preprint arXiv:1905.12616. find our card
  37. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2019. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777. find our card
  38. Shiyue Zhang and Mohit Bansal. 2019. Addressing semantic drift in question generation for semi-supervised question answering. arXiv preprint arXiv:1909.06356. find
  39. Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In EMNLP. find our card
  40. Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. In National CCF Conference on Natural Language Processing and Chinese Computing, pages 662–671. find
  41. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qi-2020-prophetnet-predicting-future-n-gram/