Elamur Research⌕
☰
home / papers / 2020 / Longformer: The Long-Document Transformer

Longformer: The Long-Document Transformer

Summary

The standard Transformer handles long texts poorly. Computation and memory for self-attention grow quadratically with sequence length. As a result, models such as BERT are limited to 512 tokens, and longer documents have to be truncated or split into chunks. The authors propose Longformer, an architecture in which full attention is replaced by a combination of two mechanisms. The first is a local sliding window, including its sparse (dilated) variant. The second is global attention at a few positions chosen for the specific task. Thanks to this, costs grow only linearly. The new mechanism can be built into an already trained model. The authors continued MLM pretraining from the RoBERTa checkpoint and then fine-tuned the model on classification, QA, and coreference resolution. For generative tasks with long inputs, they created LED, an encoder-decoder variant initialized from BART. At the time of publication, Longformer achieved the best results on text8, enwik8, WikiHop, and TriviaQA, and LED on summarization of arXiv papers. Thus, documents of thousands of tokens can be processed in a single pass, without cumbersome task-specific architectures.

Key points

  • In character-level language modeling, the model processes sequences of up to 32K characters on ordinary GPUs and achieves state-of-the-art results on text8 and enwik8.
  • In the ablation on WikiHop (dev), the base model with length 4,096 has an accuracy of 73.8. Removing global attention and its separate projections drops accuracy to 65.5 (−8.3). Restricting to 512 tokens with full attention yields 71.7, meaning the gain is not explained by additional pretraining alone.
  • In LED, the positional embeddings are extended from 1K to 16K tokens: BART's 1K embeddings were copied 16 times. There are two versions: LED-base (6 layers each in the encoder and decoder) and LED-large (12 each).
  • On the arXiv dataset, where the 90th percentile of document length is 14.5K tokens, LED-large (16,384) achieved ROUGE-1/2/L of 46.63 / 19.62 / 41.83. This is slightly better than BigBird (4,096) and Pegasus.
  • The code and models are open-sourced by the Allen Institute for AI: github.com/allenai/longformer. The authors note that BigBird later improved results on the leaderboards, but its pretraining required roughly 16 times more compute.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cite

Cite as Iz Beltagy, Matthew E. Peters and Arman Cohan. 2020. Longformer: The Long-Document Transformer. arXiv:2004.05150.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2004.05150 · arXiv 2004.05150

@misc{beltagy2020longformer,
  title = {{Longformer: The Long-Document Transformer}},
  author = {Iz Beltagy, Matthew E. Peters and Arman Cohan},
  year = {2020},
  doi = {10.48550/arxiv.2004.05150},
  eprint = {2004.05150},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2004.05150}
}

References

60 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. ETC: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processi… our card
  2. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2018. Character-level language modeling with deeper self-attention. In AAAI. find
  3. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL. our card
  4. Jifan Chen, Shih-Ting Lin, and Greg Durrett. 2019. Multi-hop question answering via reasoning chains. arXiv preprint, abs/1910.02610. — [arXiv:1910.02610](https://arxiv.org/abs/1910.02610) find
  5. Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. 2018. TVM: An automated end-to-end optimizing compiler for deep learning. In OSDI. our card
  6. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint, abs/1604.06174. find
  7. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint, abs/1904.10509. find
  8. Christopher Clark and Matt Gardner. 2017. Simple and effective multi-paragraph reading comprehension. In ACL. our card
  9. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does bert look at? an analysis of bert’s attention. arXiv preprint, abs/1906.04341. find
  10. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In NAACL-HLT 2018. our card
  11. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NeurIPS. find
  12. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In ACL. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. our card
  14. Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2020. Hierarchical graph network for multi-hop question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8823–8838, Online. Association for Computational L… find
  15. Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. find
  16. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In MRQA workshop at EMNLP. our card
  17. Alexios Gidiotis and Grigorios Tsoumakas. 2020. A divide-and-conquer approach to the summarization of academic articles. ArXiv, abs/2004.06190. — [arXiv:2004.06190](https://arxiv.org/abs/2004.06190) find
  18. Michael Glaß, Alfio Massimiliano Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, Gaudani Bhargav, Dinesh Garg, and Avirup Sil. 2019. Span selection pre-training for question answering. arXiv preprint, abs/1909.04120. find
  19. Scott Gray, Alec Radford, and Diederik P. Kingma. 2017. Gpu kernels for block-sparse weights. find
  20. Dirk Groeneveld, Tushar Khot, Mausam, and Ashish Sabhwaral. 2020. A simple yet strong pipeline for HotpotQA. arXiv preprint, abs/2004.06753. find
  21. Ankit Gupta and Jonathan Berant. 2020. Gmat: Global memory augmentation for transformers. ArXiv, abs/2006.03274. — [arXiv:2006.03274](https://arxiv.org/abs/2006.03274) find
  22. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. find
  23. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL. our card
  24. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In EMNLP-IJCNLP. our card
  25. Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. SemEval-2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829–839, Minneapolis, Minnesota, USA… our card
  26. Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. ICLR. our card
  27. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In ICLR. our card
  28. Olga V. Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of bert. In EMNLP/IJCNLP. our card
  29. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In NAACL. our card
  30. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… our card
  31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint, abs/1907.11692. find
  32. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon… our card
  33. Matt Mahoney. 2009. Large text compression benchmark. find
  34. Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. In SSW. our card
  35. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations. our card
  36. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. our card
  37. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguisti… our card
  38. Jiezhong Qiu, Hao Ma, Omer Levy, Scott Yih, Sinong Wang, and Jie Tang. 2019. Blockwise self-attention for long document understanding. arXiv preprint, abs/1911.02972. find
  39. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  40. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In ICLR. our card
  41. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67. our card
  42. Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2020. Efficient content-based sparse attention with routing transformers. arXiv preprint, abs/2003.05997. our card
  43. Nan Shao, Yiming Cui, Ting Liu, Shijin Wang, and Guoping Hu. 2020. Is graph structure necessary for multi-hop reasoning? arXiv preprint, abs/2004.03096. — [arXiv:2004.03096](https://arxiv.org/abs/2004.03096) find
  44. Sandeep Subramanian, Raymond Li, Jonathan Pilault, and C. Pal. 2020. On extractive and abstractive neural document summarization with transformer language models. In EMNLP. our card
  45. Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. In ACL. our card
  46. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
  47. Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. arXiv preprint, abs/1806.02847. find
  48. Ming Tu, Jinke Huang, Xiaodong He, and Bowen Zhou. 2020. Graph sequential network for reasoning over sequences. In NeurIPS Graph Representation Learning workshop. find
  49. Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bufang Zhou. 2019. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. arXiv preprint, abs/1911.00484. — [arXiv:1911.00484](https://arxiv.org/abs/1911.00484) find
  50. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. our card
  51. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. TACL, 6:287–302. our card
  52. Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. arXiv preprint, abs/1901.10430. our card
  53. Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation for consistency training. arXiv preprint, abs/1904.12848. our card
  54. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shu xin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Li-Wei Wang, and Tie-Yan Liu. 2020. On layer normalization in the transformer architecture. arXiv preprint, abs/2002.04745. find
  55. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP. our card
  56. Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. 2019. BP-Transformer: Modelling long-range context via binary partitioning. arXiv preprint, abs/1911.04070. find
  57. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, C. Alberti, S. Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, L. Yang, and A. Ahmed. 2020. Big bird: Transformers for longer sequences. ArXiv, abs/2007.14062. our card
  58. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS. our card
  59. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. ICML. our card
  60. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. ICCV, pages 19–27. our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/