Elamur Research
☰
home / papers / 2020 / BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.703.pdf

Summary

Earlier self-supervised pre-training methods were usually built either for understanding tasks, like BERT-style masked models, or for generation, like GPT-style left-to-right models. Few did both well. The authors propose BART, a sequence-to-sequence Transformer pre-trained as a denoising autoencoder. Input documents are corrupted by a noising function, and a bidirectional encoder works with an autoregressive decoder to restore the original text. The encoder input does not need to line up with the decoder output, so any kind of corruption can be used, including changes in length. The authors ran a controlled comparison of noising strategies and re-implemented earlier objectives. The best overall mix was shuffling sentences together with text infilling, where spans of varying length are each replaced by one mask token. After fine-tuning, the large model performs about as well as RoBERTa on GLUE and SQuAD. It also sets new state-of-the-art results in abstractive summarization, dialogue and generative question answering. This matters because one pre-trained model can handle both comprehension and generation, and the authors released code and weights publicly.

Key points

  • Architecture: a standard Transformer encoder-decoder with GeLU activations. The base model has 6 encoder and 6 decoder layers (hidden size 768), and the large model has 12 of each. It has about 10% more parameters than a BERT model of the same size.
  • Noising schemes tested: token masking, token deletion, text infilling, sentence permutation and document rotation. In text infilling, span lengths come from a Poisson distribution (λ = 3), and zero-length spans are allowed. Unlike SpanBERT, each span becomes a single [MASK].
  • Generation results: gains of up to 3.5 ROUGE on XSum, plus new top scores on several dialogue, QA and summarization benchmarks. The main weakness the authors note is a tendency to hallucinate unsupported content.
  • Machine translation: a new, randomly initialized source encoder replaces BART's embedding layer and is trained in two stages. This gives +1.1 BLEU over a back-translation baseline on WMT Romanian-English, using only English (target-side) pre-training.
  • Release: code and pre-trained models are available in fairseq and Hugging Face Transformers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

4507 papers of our library cite it (of the 140673 whose bibliographies we read); 143 of them have a page here, the first 100 below, 4364 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.703 · arXiv 1910.13461

@inproceedings{lewis2020bart,
  title = {{BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension}},
  author = {Lewis, Mike and Liu, Yinhan and Goyal, Naman and Ghazvininejad, Marjan and Mohamed, Abdelrahman and Levy, Omer and Stoyanov, Veselin and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.703},
  eprint = {1910.13461},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.703}
}

References

26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Llu’is M‘arquez, and Richard Wicentowski (eds.). Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007). Association for Computational Linguistics, Prague, Czech Republic, June 2007. find
  2. Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pp. 177– 190. Springer, 2006. find
  3. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. arXiv preprint arXiv:1907.09190, 2019. find our card
  4. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. find
  5. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pp. 1693–1701, 2015. find our card
  6. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529, 2019. find our card
  7. Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019. find
  8. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019. find
  9. Hector J Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, pp. 47, 2011. find
  10. Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345, 2019. find
  11. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging pre-trained checkpoints for sequence generation tasks. arXiv preprint arXiv:1907.12461, 2019. find our card
  12. Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368, 2017. find
  13. Rico Sennrich, Barry Haddow, and Alexandra Birch. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, 2016. find our card
  14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. find
  15. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pp. 1631–1642, 2013. find
  16. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. find
  17. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mass: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning, 2019. find our card
  18. Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018. find
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017. find
  20. Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017. find our card
  21. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. find our card
  22. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. find
  23. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint 1805.12471, 2018. find
  24. Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for Alec Radford, Karthik Narasimhan, Tim Salimans, sentence understanding through inference. arXiv and Ilya Sutskever. Improving language un-preprint arXiv:1704.05426, 2017. derstanding by generative pre-training.… find
  25. OpenAI Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. find
  26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. find our card