🔗 Who cites «Generating Long Sequences with Sparse Transformers»

543 papers of our library cite it (of the 149,439 whose bibliographies we read) · 86 with a page here, the most cited first · ← back to «Generating Long Sequences with Sparse Transformers»

1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScalearXiv:2010.11929Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al.2020 · 23,726 citations
2. Language Models are Few-Shot LearnersarXiv:2005.14165Tom B. Brown, Benjamin Mann, Nick Ryder, et al.2020 · 14,178 citations
3. PaLM: Scaling Language Modeling with PathwaysarXiv:2204.02311Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, et al.2022 · 4,753 citations
4. Longformer: The Long-Document TransformerarXiv:2004.05150Iz Beltagy, Matthew E. Peters, Arman Cohan2020 · 3,880 citations
5. Scaling Laws for Neural Language ModelsarXiv:2001.08361Jared Kaplan, Sam McCandlish, Tom Henighan, et al.2020 · 3,434 citations
6. On the Opportunities and Risks of Foundation ModelsarXiv:2108.07258Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, et al.2021 · 3,160 citations
7. A Survey of Large Language ModelsFrontiers of Computer ScienceWayne Xin Zhao, Kun Zhou, Junyi Li, et al.2023 · 2,875 citations
8. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsarXiv:1909.11942Zhenzhong Lan, Mingda Chen, Sebastian Goodman, et al.2019 · 2,806 citations
9. Scalable Diffusion Models with TransformersIEEE/CVF International Conference on Computer Vision (ICCV)William Peebles, Saining Xie2022 · 2,670 citations
10. Evaluating Large Language Models Trained on CodearXiv:2107.03374Mark Chen, Jerry Tworek, Heewoo Jun, et al.2021 · 2,659 citations
11. Robust Speech Recognition via Large-Scale Weak SupervisionarXiv:2212.04356Alec Radford, Jong Wook Kim, Tao Xu, et al.2022 · 2,546 citations
12. DeBERTa: Decoding-enhanced BERT with Disentangled AttentionarXiv:2006.03654Pengcheng He, Xiaodong Liu, Jianfeng Gao, et al.2020 · 1,828 citations
13. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelsarXiv:2206.04615 · Transactions on Machine Learning Research, May/2022, https://openreview.net/forum?id=uyTL5BvosjAarohi Srivastava, Abhinav Rastogi, Abhishek Rao, et al.2022 · 1,756 citations
14. Efficient Transformers: A SurveyACM Computing SurveysYi Tay, Mostafa Dehghani, Dara Bahri, et al.2020 · 1,539 citations
15. Zero-Shot Text-to-Image GenerationarXiv:2102.12092Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, et al.2021 · 1,528 citations
16. CTRL: A Conditional Transformer Language Model for Controllable GenerationarXiv:1909.05858Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, et al.2019 · 1,291 citations
17. A Comprehensive Overview of Large Language ModelsACM Transactions on Intelligent Systems and TechnologyHumza Naveed, Asad Ullah Khan, Shi Qiu, et al.2023 · 1,287 citations
18. Big Bird: Transformers for Longer SequencesConference on Neural Information Processing Systems (NeurIPS)Manzil Zaheer, Guru Guruganesh, Avinava Dubey, et al.2020 · 1,023 citations
19. GPT-NeoX-20B: An Open-Source Autoregressive Language ModelBigScience Episode – Workshop on Challenges & Perspectives in Creating Large Language ModelsSidney Black, Stella Biderman, Eric Hallahan, et al.2022 · 831 citations
20. Reformer: The Efficient TransformerarXiv:2001.04451Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya2020 · 829 citations
21. Can Large Language Models Transform Computational Social Science?Computational LinguisticsCaleb Ziems, William Held, Omar Shaikh, et al.2023 · 790 citations
22. Efficient Content-Based Sparse Attention with Routing TransformersTransactions of the Association for Computational Linguistics (TACL)Aurko Roy, Mohammad Saffar, Ashish Vaswani, et al.2020 · 677 citations
23. gpt-oss-120b & gpt-oss-20b Model CardarXiv:2508.10925OpenAI: Sandhini Agarwal, Lama Ahmad, Jason Ai, et al.2025 · 576 citations
24. LongBench: A Bilingual, Multitask Benchmark for Long Context UnderstandingarXiv:2308.14508Yushi Bai, Xin Lv, Jiajie Zhang, et al.2023 · 514 citations
25. Rethinking Attention with PerformersarXiv:2009.14794Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, et al.2020 · 491 citations
26. BERT for Coreference Resolution: Baselines and AnalysisConference on Empirical Methods in Natural Language Processing (EMNLP)Mandar Joshi, Omer Levy, Luke Zettlemoyer, et al.2019 · 490 citations
27. Analyzing the Structure of Attention in a Transformer Language ModelarXiv:1906.04284Jesse Vig, Yonatan Belinkov2019 · 438 citations
28. Long Range Arena: A Benchmark for Efficient TransformersarXiv:2011.04006Yi Tay, Mostafa Dehghani, Samira Abnar, et al.2020 · 414 citations
29. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisionConference on Neural Information Processing Systems (NeurIPS)Jay Shah, Ganesh Bikshandi, Ying Zhang, et al.2024 · 400 citations
30. RULER: What's the Real Context Size of Your Long-Context Language Models?arXiv:2404.06654Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, et al.2024 · 400 citations
31. A Simple Language Model for Task-Oriented DialoguearXiv:2005.00796Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, et al.2020 · 399 citations
32. ETC: Encoding Long and Structured Inputs in TransformersConference on Empirical Methods in Natural Language Processing (EMNLP)Joshua Ainslie, Santiago Ontañón, Chris Alberti, et al.2020 · 386 citations
33. Compressive Transformers for Long-Range Sequence ModellingarXiv:1911.05507Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, et al.2019 · 340 citations
34. Efficient Attentions for Long Document SummarizationConference of the North American Chapter of the Association for Computational Linguistics (NAACL)Luyang Huang, Shuyang Cao, Nikolaus Parulian, et al.2021 · 336 citations
35. Structured Denoising Diffusion Models in Discrete State-SpacesarXiv:2107.03006Jacob Austin, Daniel D. Johnson, Jonathan Ho, et al.2021 · 284 citations
36. Scaling Laws for Autoregressive Generative ModelingarXiv:2010.14701Tom Henighan, Jared Kaplan, Mor Katz, et al.2020 · 277 citations
37. Adaptively Sparse TransformersConference on Empirical Methods in Natural Language Processing (EMNLP)Gonçalo M. Correia, Vlad Niculae, André F. T. Martins2019 · 275 citations
38. Perceiver: General Perception with Iterative AttentionarXiv:2103.03206Andrew Jaegle, Felix Gimeno, Andrew Brock, et al.2021 · 269 citations
39. LongLoRA: Efficient Fine-tuning of Long-Context Large Language ModelsarXiv:2309.12307Yukang Chen, Shengju Qian, Haotian Tang, et al.2023 · 268 citations
40. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI ScalearXiv:2201.05596Samyam Rajbhandari, Conglong Li, Zhewei Yao, et al.2022 · 261 citations
41. TrustLLM: Trustworthiness in Large Language ModelsarXiv:2401.05561Yue Huang, Lichao Sun, Haoran Wang, et al.2024 · 237 citations
42. Compressing Context to Enhance Inference Efficiency of Large Language ModelsConference on Empirical Methods in Natural Language Processing (EMNLP)Yucheng Li, Bo Dong, Frank Guerin, et al.2023 · 212 citations
43. mGPT: Few-Shot Learners Go MultilingualarXiv:2204.07580Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, et al.2022 · 203 citations
44. MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse AttentionConference on Neural Information Processing Systems (NeurIPS)Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, et al.2024 · 202 citations
45. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMsarXiv:2310.01801Suyu Ge, Yunan Zhang, Liyuan Liu, et al.2023 · 202 citations
46. Effective Long-Context Scaling of Foundation ModelsarXiv:2309.16039Wenhan Xiong, Jingyu Liu, Igor Molybog, et al.2023 · 184 citations
47. Adapting Language Models to Compress ContextsConference on Empirical Methods in Natural Language Processing (EMNLP)Alexis Chevalier, Alexander Wettig, Anirudh Ajith, et al.2023 · 182 citations
48. Learning to Compress Prompts with Gist TokensConference on Neural Information Processing Systems (NeurIPS)Jesse Mu, Xiang Lisa Li, Noah Goodman2023 · 178 citations
49. Generative Representational Instruction TuningarXiv:2402.09906Niklas Muennighoff, Hongjin Su, Liang Wang, et al.2024 · 172 citations
50. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer ModelsarXiv:2309.14509Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, et al.2023 · 162 citations
51. Efficient Methods for Natural Language Processing: A SurveyTransactions of the Association for Computational Linguistics (TACL)Marcos Treviso, Ji-Ung Lee, Tianchu Ji, et al.2023 · 145 citations
52. Retrieval meets Long Context Large Language ModelsarXiv:2310.03025Peng Xu, Wei Ping, Xianchao Wu, et al.2023 · 130 citations
53. InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context MemoryConference on Neural Information Processing Systems (NeurIPS)Chaojun Xiao, Pengle Zhang, Xu Han, et al.2024 · 126 citations
54. A Length-Extrapolatable TransformerAnnual Meeting of the Association for Computational Linguistics (ACL)Yutao Sun, Li Dong, Barun Patra, et al.2023 · 118 citations
55. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming HeadsarXiv:2410.10819Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, et al.2024 · 114 citations
56. Rethinking Document-level Neural Machine TranslationFindings of the Association for Computational LinguisticsZewei Sun, Mingxuan Wang, Hao Zhou, et al.2022 · 107 citations
57. Do Long-Range Language Models Actually Use Long-Range Context?Conference on Empirical Methods in Natural Language Processing (EMNLP)Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, et al.2021 · 101 citations
58. CoLT5: Faster Long-Range Transformers with Conditional ComputationConference on Empirical Methods in Natural Language Processing (EMNLP)Joshua Ainslie, Tao Lei, Michiel de Jong, et al.2023 · 84 citations
59. MoBA: Mixture of Block Attention for Long-Context LLMsConference on Neural Information Processing Systems (NeurIPS)Enzhe Lu, Zhejun Jiang, Jingyuan Liu, et al.2025 · 80 citations
60. What Context Features Can Transformer Language Models Use?Annual Meeting of the Association for Computational Linguistics (ACL)Joe O'Connor, Jacob Andreas2021 · 77 citations
61. Conditional Generation with a Question-Answering BlueprintTransactions of the Association for Computational Linguistics (TACL)Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, et al.2023 · 70 citations
62. The Devil in Linear TransformerConference on Empirical Methods in Natural Language Processing (EMNLP)Zhen Qin, Xiaodong Han, Weixuan Sun, et al.2022 · 67 citations
63. A Comprehensive Survey on Long Context Language ModelingarXiv:2503.17407Jiaheng Liu, Dawei Zhu, Zhiqi Bai, et al.2025 · 64 citations
64. XAttention: Block Sparse Attention with Antidiagonal ScoringarXiv:2503.16428Ruyi Xu, Guangxuan Xiao, Haofeng Huang, et al.2025 · 59 citations
65. BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language ModelsLanguage Resources and Evaluation Conference (LREC)Zican Dong, Tianyi Tang, Junyi Li, et al.2024 · 59 citations
66. Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language ModelingarXiv:2406.07522Liliang Ren, Yang Liu, Yadong Lu, et al.2024 · 59 citations
67. Simple linear attention language models balance the recall-throughput tradeoffarXiv:2402.18668Simran Arora, Sabri Eyuboglu, Michael Zhang, et al.2024 · 59 citations
68. RecurrentGPT: Interactive Generation of (Arbitrarily) Long TextarXiv:2305.13304Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, et al.2023 · 57 citations
69. Zoology: Measuring and Improving Recall in Efficient Language ModelsarXiv:2312.04927Simran Arora; Sabri Eyuboglu; Aman Timalsina; Isys Johnson; Michael Poli; James Zou; Atri Rudra;, Christopher Re2023 · 55 citations
70. Incremental Neural Coreference Resolution in Constant MemoryPatrick Xia, João Sedoc, Benjamin Van Durme2020 · 55 citations
71. MiniMax-01: Scaling Foundation Models with Lightning AttentionarXiv:2501.08313MiniMax, Aonian Li, Bangwei Gong, et al.2025 · 50 citations
72. TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormerarXiv:2307.14995Zhen Qin, Dong Li, Weigao Sun, et al.2023 · 33 citations
73. Linear algebra with transformersarXiv:2112.01898François Charton2021 · 31 citations
74. MEMORYLLM: Towards Self-Updatable Large Language ModelsarXiv:2402.04624Yu Wang, Yifan Gao, Xiusi Chen, et al.2024 · 27 citations
75. ERNIE-Doc: A Retrospective Long-Document Modeling TransformerarXiv:2012.15688Siyu Ding, Junyuan Shang, Shuohuan Wang, et al.2020 · 27 citations
76. Block-State TransformersarXiv:2306.09539Mahan Fathi, Jonathan Pilault, Orhan Firat, et al.2023 · 20 citations
77. Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language ModelsAnnual Meeting of the Association for Computational Linguistics (ACL)Zhisong Zhang, Yan Wang, Xinting Huang, et al.2025 · 14 citations
78. PanGu-π: Enhancing Language Model Architectures via Nonlinearity CompensationarXiv:2312.17276Yunhe Wang, Hanting Chen, Yehui Tang, et al.2023 · 10 citations
79. GRIN: GRadient-INformed MoEarXiv:2409.12136Liyuan Liu, Young Jin Kim, Shuohang Wang, et al.2024 · 7 citations
80. DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context ReasoningFindings of the Association for Computational LinguisticsHossein Entezari Zarch, Lei Gao, Chaoyi Jiang, et al.2026 · 1 citation
81. AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache ReuseAnnual Meeting of the Association for Computational Linguistics (ACL)Jie Ou, Jinyu Guo, Shiyao Guo, et al.2026
82. From Logical to Computational Sparsity: Structure-Aware Block-Sparse Attention for Long-Code CompletionAnnual Meeting of the Association for Computational Linguistics (ACL)Yanli Wang, Yanlin Wang, Bowen Zhang, et al.2026
83. Less is More: Knowledge-Aware Compression for Long Legal Judgment PredictionFindings of the Association for Computational LinguisticsFanghao Lou, Qiqi Wang, Guanyu Chen, et al.2026
84. MATCH: Modulating Attention via In-Context Retrieval for Long-Context TransformersAnnual Meeting of the Association for Computational Linguistics (ACL)Linrui Ma, Chun Hei Lo, Xinyu Wang, et al.2026
85. RoBSA: RoPE-based Blockwise Sparse Multi-head Latent AttentionAnnual Meeting of the Association for Computational Linguistics (ACL)Xinyu Shi, Kairong Luo, Zhen Zheng, et al.2026
86. SCOPE: Boosting LLM Efficiency with Scoped Position EncodingAnnual Meeting of the Association for Computational Linguistics (ACL)Qingguo Qi, Hongyang Chen, Zhao Li2026

457 more papers of our library cite it; they have no page here yet.

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/cited-by/