Elamur Research
☰
home / papers / 2020 / Long Range Arena: A Benchmark for Efficient Transformers

Long Range Arena: A Benchmark for Efficient Transformers

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2011.04006

Summary

Many "efficient" Transformer variants (xformers) have been proposed to get around the quadratic cost of self-attention. Each one was tested on its own choice of tasks, often mixed with pretraining, so they could not be fairly compared. The authors built Long-Range Arena (LRA) to fix this. It is a single, lightweight benchmark of encoder-only classification tasks with inputs of 1K to 16K tokens. The tasks cover several data types: hierarchical math expressions, byte-level text, document pairs, pixel sequences from images, and synthetic visual path puzzles. Pretraining and data augmentation are not allowed, so differences in scores reflect the architecture itself. Ten efficient models plus the vanilla Transformer were trained under shared settings in JAX/Flax and compared on accuracy, speed and memory. No model was best on every task. Every model failed the hardest 16K-token task, which shows that long-range modelling is still an open problem. Because it is open-source and cheap to run, LRA became a standard testbed for long-sequence architectures.

Key points

  • The six tasks are Long ListOps (up to 2K tokens, 10-way classification), byte-level IMDb sentiment classification (4K), byte-level ACL Anthology Network (AAN) citation matching in a two-tower setup with no cross-attention (4K per document, 8K total), CIFAR-10 as grayscale pixel sequences (1024), Pathfinder (32×32, so 1024), and Pathfinder-X (16K).
  • The models evaluated include Sparse Transformer, Reformer, Linformer, Longformer, Sinkhorn Transformer, Performer (FAVOR+), Synthesizer, Linear Transformer, BigBird and local attention.
  • No model could learn Path-X, whether trained from scratch or transferred from Pathfinder.
  • Byte-level text is much harder than word-level: un-pretrained models drop from the high 80s to the mid 60s in accuracy on IMDb.
  • Under the fair-comparison rules, a new model can only be compared against the published table if it keeps the same model size, uses no pretraining, and keeps the task setups unchanged. Code: github.com/google-research/long-range-arena.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

209 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 200 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2011.04006 · arXiv 2011.04006

@misc{tay2020long,
  title = {{Long Range Arena: A Benchmark for Efficient Transformers}},
  author = {Tay, Yi and Dehghani, Mostafa and Abnar, Samira and Shen, Yikang and Bahri, Dara and Pham, Philip and Rao, Jinfeng and Yang, Liu and Ruder, Sebastian and Metzler, Donald},
  year = {2020},
  doi = {10.48550/arxiv.2011.04006},
  eprint = {2011.04006},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2011.04006}
}

References

51 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. find our card
  2. Joshua Ainslie, Santiago Ontanon, Chris Alberti, Philip Pham, Anirudh Ravula, and Sumit Sanghai. Etc: Encoding long and structured data in transformers. arXiv preprint arXiv:2004.08483, 2020. 5 Datasets such as LAMBADA (Paperno et al., 2016) more explicitly test for context understanding but are sti… — [arXiv:2004.08483](https://arxiv.org/abs/2004.08483) find
  3. Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. find
  4. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. find
  5. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Jared Davis, Tamas Sarlos, David Belanger, Lucy Colwell, and Adrian Weller. Masked language modeling for proteins via linearly scalable long-context transformers. arXiv preprint arXiv:2006.03555, 2020a. — [arXiv:2006.03555](https://arxiv.org/abs/2006.03555) find
  6. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020b. find
  7. Zihang Dai, Zhilin Yang, Yiming Yang, William W. Cohen, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. In Proceedings of ACL 2019, 2019. find
  8. Michał Daniluk, Tim Rockt, Johannes Welbl, and Sebastian Riedel. Frustratingly Short Attention Spans in Neural Language Modeling. In Proceedings of ICLR 2017, 2017. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  10. Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 55–64, 2016. find
  11. Kelvin Guu, Kenton Lee, Zora Tung, and Panupong Pasupat. REALM: Retrieval-Augmented Language Model Pre-Training. In Proceedings of ICML 2020, 2020. find
  12. Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180, 2019. — [arXiv:1912.12180](https://arxiv.org/abs/1912.12180) find
  13. Sara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489, 2020. — [arXiv:2009.06489](https://arxiv.org/abs/2009.06489) find
  14. R. Houtkamp and P. R. Roelfsema. Parallel and serial grouping of image elements in visual perception. J Exp Psychol Hum Percept Perform,, 2010. find
  15. Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. In Proceedings of ACL 2018, 2018. find
  16. Jyun-Yu Jiang, Mingyang Zhang, Cheng Li, Michael Bendersky, Nadav Golbandi, and Marc Najork. Semantic text matching for long-form documents. In The World Wide Web Conference, pp. 795– 806, 2019. find
  17. Mandar Joshi, Eunsol Choi, Daniel S Weld, Luke Zettlemoyer, and Paul G Allen. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of ACL 2017, 2017. find our card
  18. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. arXiv preprint arXiv:2006.16236, 2020. find
  19. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. Learning to discover, ground and use words with segmental neural language models. In Proceedings of ACL 2019, pp. 6429–6441, 2019. find
  20. Junkyung Kim*, Drew Linsley*, Kalpit Thakkar, and Thomas Serre. Disentangling neural mechanisms for perceptual grouping. In International Conference on Learning Representations, 2020. find
  21. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020. URL https://openreview. net/forum?id=rkgNKkHtvB. find our card
  22. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA Reading Comprehension Challenge. Transactions of the Association for Computational Linguistics, 2018. find our card
  23. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009. find
  24. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin Kenton Lee, Kristina Toutanova, Llion Jones Matthew Kelcey, Ming-Wei Chang, Andrew M Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questions… find our card
  25. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent Retrieval for Weakly Supervised Open Domain Question Answering. In Proceedings of ACL 2019, 2019. find our card
  26. Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. In Advances in neural information processing systems, pp. 152–164, 2018. find
  27. Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198, 2018. find our card
  28. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, pp. 13–23, 2019. find our card
  29. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150, Portland, Oregon, USA, J… find our card
  30. Nikita Nangia and Samuel R Bowman. Listops: A diagnostic dataset for latent tree learning. arXiv preprint arXiv:1804.06028, 2018. find
  31. Denis Paperno, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of ACL 2016, 2016. find our card
  32. Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. arXiv preprint arXiv:1802.05751, 2018. — [arXiv:1802.05751](https://arxiv.org/abs/1802.05751) find
  33. Dragomir R Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara. The acl anthology network corpus. Language Resources and Evaluation, 47(4):919–944, 2013. find
  34. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. 2019. find
  35. Jack W Rae and Ali Razavi. Do Transformers Need Deep Long-Range Memory? In Proceedings of ACL 2020, pp. 7524–7529, 2020. find
  36. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. find
  37. Alexander Rives, Siddharth Goyal, Joshua Meier, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. bioRxiv, pp. 622803, 2019. find
  38. Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. arXiv preprint arXiv:2003.05997, 2020. find our card
  39. Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 15–18, 2019. find our card
  40. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP 2013, pp. 1631–1642. find
  41. Hao Tan and Mohit Bansal. LXMERT: Learning Cross-Modality Encoder Representations from Transformers. In Proceedings of EMNLP 2019, 2019. find our card
  42. Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. arXiv preprint arXiv:2005.00743, 2020a. — [arXiv:2005.00743](https://arxiv.org/abs/2005.00743) find
  43. Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. arXiv preprint arXiv:2002.11296, 2020b. — [arXiv:2002.11296](https://arxiv.org/abs/2002.11296) find
  44. Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprint arXiv:2009.06732, 2020c. find our card
  45. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017. find
  46. Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. find
  47. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing Datasets for Multi-hop Reading Comprehension Across Documents. In Transactions of the Association for Computational Linguistics, 2018. find our card
  48. Adina Williams, Nikita Nangia, and Samuel R. Bowman. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of NAACL-HLT 2018, 2018. URL http://arxiv.org/abs/1704.05426. ↗
  49. Liu Yang, Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. Beyond 512 tokens: Siamese multi-depth transformer-based hierarchical encoder for document matching. CoRR, abs/2004.12297, 2020. URL https://arxiv.org/abs/2004.12297. — [arXiv:2004.12297](https://arxiv.org/abs/2004.12297) ↗
  50. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of EMNLP 2018, 2018. find our card
  51. Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.14062, 2020. find our card