Elamur Research⌕
☰
home / papers / 2020 / ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation

ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation

Paper · 2020 · Dongling Xiao, Han Zhang, Yukun Li et al.

Summary

ERNIE-GEN, from Baidu, is a sequence-to-sequence framework that covers both pre-training and fine-tuning for text generation tasks. It targets exposure bias: during training the decoder conditions on gold tokens, but at inference it conditions on its own outputs, so early errors propagate. The model uses a Multi-Flow Attention design built on a Transformer whose parameters are shared by encoder and decoder. It has three main components:

  • Infilling generation: an inserted [ATTN] symbol, rather than the previous word, collects context at each step.
  • Noise-aware generation: target tokens are randomly replaced with arbitrary vocabulary words, which teaches the model to tolerate its own mistakes.
  • Span-by-span flow: the model predicts whole semantic spans, running in parallel with word-by-word prediction.

Pre-training data come from unlabeled text, where multiple fragments of varying length are sampled as targets. The authors report state-of-the-art results on summarization, question generation, dialogue and generative QA, with less pre-training data and fewer parameters than competing models. Code and models are publicly released.

Key points

  • Benchmarks: Gigaword and CNN/DailyMail (summarization), SQuAD (question generation), Persona-Chat (dialogue) and CoQA (generative QA). Code is available at github.com/PaddlePaddle/ERNIE.
  • Target sampling: fragments cover about 25% of the input (γ = 0.25). Fragment lengths are drawn from U(1,4) with probability 0.4 or U(4,32) with probability 0.6.
  • Ablation (ERNIE-GEN BASE, Gigaword 10k, ROUGE-1/2/L):
    • Full model: 33.75 / 15.23 / 31.35.
    • Without both noise-aware generation and span-by-span generation: 33.23 / 14.77 / 31.00.
  • Noise-aware fine-tuning vs. masking: predicting every decoder token under noise outperformed UniLM-style masked-word fine-tuning (masking probability 0.7).
  • Attention analysis: higher noise rates led the model to give lower attention weights to corrupted target tokens.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢BaiduBeijing · China40 works · 14,627 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksText generation · 145 works

Cite

Cite as Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu and Haifeng Wang. 2020. ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation. arXiv:2001.11314.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2001.11314 · arXiv 2001.11314

@misc{xiao2020ernie,
  title = {{ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation}},
  author = {Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu and Haifeng Wang},
  year = {2020},
  doi = {10.48550/arxiv.2001.11314},
  eprint = {2001.11314},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2001.11314}
}

References

30 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. Plato: Pre-trained dialogue generation model with discrete latent variable. In ACL, 2020.
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186, 2019.
  3. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiaowuen Hon. Unified language model pre-training for natural language understanding and generation. In NIPS, pages 13042–13054, 2019.
  4. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, pages 1693–1701, 2015.
  5. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529, 2019.
  6. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radue Soricut. Albert: A lite bert for self-supervised learning of language representations. In ICLR, 2020. find
  7. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
  8. Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In EMNLP-IJCNLP, pages 3721–3731, 2019. find
  9. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. find
  10. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, pages 2227–2237, 2018.
  11. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. find
  12. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.
  13. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, pages 2383–2392, 2016.
  14. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2016.
  15. Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. In ACL, pages 249–266, 2019.
  16. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging pre-trained checkpoints for sequence generation tasks. arXiv preprint arXiv:1907.12461, 2019.
  17. Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In EMNLP, 2015. find
  18. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mass: Masked sequence to sequence pre-training for language generation. In ICML, 2019.
  19. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223, 2019.
  20. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding. In AAAI, 2020.
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998–6008, 2017.
  22. Li Wang, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du. A reinforced topic-aware convolutional sequence-to-sequence model for abstractive text summarization. In IJCAI, 2018. find
  23. Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. Topic-guided variational autoencoder for text generation. In NAACL-HLT, 2019. find
  24. Qian Yang, Dinghan Shen, Yong Cheng, Wenlin Wang, Guoyin Wang, Lawrence Carin, et al. An end-to-end generative architecture for paraphrase generation. In EMNLP-IJCNLP, pages 3123–3133, 2019. find
  25. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NIPS, pages 5754–5764, 2019.
  26. Shiyue Zhang and Mohit Bansal. Addressing semantic drift in question generation for semi-supervised question answering. In EMNLP-IJCNLP, pages 2495–2509, 2019.
  27. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? In ACL, pages 2204–2213, 2018.
  28. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint arXiv:1912.08777, 2019.
  29. Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In EMNLP, pages 3901–3910, 2018.
  30. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pages 19–27, 2015.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2020-ernie-gen-enhanced-multi/