Elamur Research⌕
☰
home / papers / 2024 / xLSTM: Extended Long Short-Term Memory

xLSTM: Extended Long Short-Term Memory

Paper · 2024 · Maximilian Beck, Korbinian Pöppel, Markus Spanring et al.

Summary

The authors examine whether the classic LSTM recurrent network, which has been displaced by Transformers, can compete with modern large language models. To this end, they address its three known weaknesses: the model cannot revise a storage decision once made, its scalar memory has limited capacity, and its computations must be performed sequentially. They propose exponential gating with a normalizer state and stabilization against numerical overflow, as well as two new cells. sLSTM has scalar memory and a new form of memory mixing within individual "heads". mLSTM stores key–value pairs in a matrix memory using a covariance update rule and is fully parallelizable during training. Both cells are embedded in residual blocks, and a stack of such blocks forms the xLSTM architecture. Models ranging from 125M to 1.3B parameters were trained on 300B tokens of SlimPajama and compared with Llama, Mamba and RWKV-4. xLSTM achieved lower perplexity, better scaling curves and linear generation time. The work shows that recurrent networks can once again be a real alternative to attention. Moreover, they use a constant amount of memory during inference.

Key points

  • The architecture has two new cell variants. sLSTM with memory mixing cannot be parallelized, but its CUDA kernel is less than twice as slow as mLSTM. mLSTM uses a d×d matrix and fully parallel training. The code is available at github.com/NX-AI/xlstm.
  • On the PALOMA benchmark with 571 text domains, xLSTM achieves lower perplexity than Llama in 486 domains (85.1%) and lower than RWKV-4 in 570 domains (99.8%).
  • Scaling laws on the SlimPajama validation set give the following ordering from worst to best model: RWKV-4, Llama, Mamba, xLSTM. The margin of xLSTM over Mamba is roughly the same as that of Mamba over Llama.
  • Inference was tested on 1.3B models on an A100-80GB. Generation time grows linearly for xLSTM and quadratically for Llama. Llama runs out of memory (OOM) already at small batch sizes, while xLSTM[1:0] has higher throughput than Mamba.
  • The authors identify the following limitations. The unoptimized mLSTM kernels are about four times slower than FlashAttention. Forget gate initialization is sensitive. The architecture and hyperparameters for large models were not fully tuned. No memory overload was observed for contexts up to 16k.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter and Sepp Hochreiter. 2024. xLSTM: Extended Long Short-Term Memory. arXiv:2405.04517.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.04517 · arXiv 2405.04517

@misc{beck2024xlstm,
  title = {{xLSTM: Extended Long Short-Term Memory}},
  author = {Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter and Sepp Hochreiter},
  year = {2024},
  doi = {10.48550/arxiv.2405.04517},
  eprint = {2405.04517},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.04517}
}

References

128 references, 43 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Achiam, S. Adler, S. Agarwal, et al. GPT-4 technical report. ArXiv, 2303.08774, 2023.
  2. J. Anderson, J. Silverstein, S. Ritz, and R. Jones. Distinctive features, categorical perception, and probability learning: Some applications of a neural model. Psychological Review, 84:413–451, 1977. doi: 10.1037/0033-295X.84.5.413. — [doi:10.1037/0033-295x.84.5.413](https://doi.org/10.1037/0033-295x.84.5.413) find
  3. J. A. Anderson. A simple neural network generating an interactive memory. Mathematical Bio-sciences, 14, 1972. doi: 10.1016/0025-5564(72)90075-2. — [doi:10.1016/0025-5564(72)90075-2](https://doi.org/10.1016/0025-5564%2872%2990075-2) find
  4. S. Arora, S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli, J. Zou, A. Rudra, and C. Ré. Zoology: Measuring and improving recall in efficient language models. ArXiv, 2312.04927, 2023.
  5. J. Ba, G. E. Hinton, V. Mnih, J. Z. Leibo, and C. Ionescu. Using fast weights to attend to the recent past. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4331–4339. Curran Associates, Inc., 2016a. find
  6. J. Ba, J. R. Kiros, and G. Hinton. Layer normalization. ArXiv, 1607.06450, 2016b.
  7. A. Bau, Y. Belinkov, H. Sajjad, N. Durrani, F. Dalvi, and J. Glass. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations (ICLR), 2019. URL https://openreview.net/forum?id=H1z-PsR5KX. ↗
  8. Y. Bisk, R. Zellers, R. LeBras, J. Gao, and Y. Choi. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, volume 34, pp. 7432–7439, 2020.
  9. S. L. Blodgett, L. Green, and B. O’Connor. Demographic dialectal variation in social media: A case study of African-American English. In Conference on Empirical Methods in Natural Language Processing, pp. 1119–1130, 2016. doi: 10.18653/v1/D16-1120.
  10. T. Brown, B. Mann, N. Ryder, et al. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877–1901. find
  11. Curran Associates, Inc., 2020. find
  12. K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlós, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, D. B. Belanger, L. J. Colwell, and A. Weller. Rethinking attention with performers. In 9th International Conference on Learning Representations (ICLR). OpenReview.net, 2021.…
  13. A. Chowdhery, S. Narang, J. Devlin, et al. PaLM: scaling language modeling with pathways. ArXiv, 2204.02311, 2022.
  14. A. Chronopoulou, M. Peters, and J. Dodge. Efficient hierarchical domain adaptation for pretrained language models. In Conference of the North American Chapter of the Association for Computational Linguistics, pp. 1336–1351, 2022. doi: 10.18653/v1/2022.naacl-main.96. find
  15. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. ArXiv, 1803.05457, 2018.
  16. T. M. Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. Electronic Computers, IEEE Transactions on, EC-14(3):326–334, 1965. find
  17. T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), volume 12, 2024. URL https://openreview.net/forum?id=mZn2Xyh9Ec. ↗
  18. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. Flashattention: Fast and memory-efficient exact attention with IO-awareness. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https://openreview.net/ forum?id=H4DqfPSibmx. ↗
  19. P. Dayan and D. J. Willshaw. Optimising synaptic learning rules in linear associative memories. Biological Cybernetics, 65, 1991. doi: 10.1007/bf00206223. — [doi:10.1007/bf00206223](https://doi.org/10.1007/bf00206223) find
  20. S. De, S. L. Smith, A. Fernando, A. Botev, G. Cristian-Muraru, A. Gu, R. Haroun, L. Berrada, Y. Chen, S. Srinivasan, G. Desjardins, A. Doucet, D. Budden, Y. W. Teh, R. Pascanu, N. DeFreitas, and C. Gulcehre. Griffin: Mixing gated linear recurrences with local attention for efficient language models.… find
  21. J. Degrave, F. Felici, J. Buchli, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602:414–419, 2022. doi: 10.1038/s41586-021-04301-9. find
  22. G. Delétang, A. Ruoss, J. Grau-Moya, T. Genewein, L. K. Wenliang, E. Catt, C. Cundy, M. Hutter, S. Legg, J. Veness, and P. A. Ortega. Neural networks and the Chomsky hierarchy. In International Conference on Learning Representations (ICLR), volume 11, 2023. URL https://openreview. net/forum?id=WbxHA…
  23. N. Du, Y. Huang, A. M. Dai, et al. GLaM: efficient scaling of language models with mixture-of-experts. ArXiv, 2112.06905, 2021.
  24. D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=COZDy0WYGg. ↗
  25. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: An 800gb dataset of diverse text for language modeling. ArXiv, 2101.00027, 2021. find
  26. F. A. Gers, J. Schmidhuber, and F. Cummins. Learning to forget: Continual prediction with LSTM. Neural Compututation, 12(10):2451–2471, 2000. find
  27. Gemini Team Google. Gemini: A family of highly capable multimodal models. ArXiv, 2312.11805, 2023.
  28. A. Graves. Generating sequences with recurrent neural networks. ArXiv, 1308.0850, 2013. find
  29. S. Greenbaum and G. Nelson. The international corpus of English (ICE) project. World Englishes, 15(1):3–15, 1996. find
  30. K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber. LSTM: A search space odyssey. ArXiv, 1503.04069, 2015.
  31. A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. ArXiv, 2312.00752, 2023. find
  32. A. Gu, K. Goel, and C. Ré. Efficiently modeling long sequences with structured state spaces. ArXiv, 2111.00396, 2021. find
  33. A. Gupta, A. Gu, and J. Berant. Diagonal state spaces are as effective as structured state spaces. ArXiv, 2203.14343, 2022. find
  34. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016.
  35. S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Master’s thesis, Technische Universität München, 1991. find
  36. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997a. find
  37. S. Hochreiter and J. Schmidhuber. LSTM can solve hard long time lag problems. In M. C. Mozer, M. I. Jordan, and T. Petsche (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 9, pp. 473–479. MIT Press, Cambridge MA, 1997b. find
  38. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In J. Kolen and S. Kremer (eds.), A Field Guide to Dynamical Recurrent Networks. IEEE, 2000. find
  39. S. Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient descent. In G. Dorffner, H. Bischof, and K. Hornik (eds.), Proc. Int. Conf. on Artificial Neural Networks (ICANN 2001), pp. 87–94. find
  40. S. Hochreiter, M. Heusel, and K. Obermayer. Fast model-based protein homology detection without alignment. Bioinformatics, 23(14):1728–1736, 2007. find
  41. J. Hoffmann, S. Borgeaud, A. Mensch, et al. Training compute-optimal large language models. ArXiv, 2203.15556, 2022.
  42. M. D. Hossain, F. Sohel, M. F. Shiratuddin, and H. Laga. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CSUR), 51(6):118, 2019. find
  43. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. ArXiv, 2001.08361, 2020.
  44. A. Karpathy. The unreasonable effectiveness of http://karpathy.github.io/2015/05/21/rnn-effectiveness/, 2015. recurrent neural networks. ↗
  45. A. Karpathy. OpenAI Five defeats Dota 2 world champions. https://openai.com/research/openai-five-defeats-dota-2-world-champions, 2019. ↗
  46. A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3128–3137, 2015.
  47. A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In E. H. Daumé III and A. Singh (eds.), International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 5156–5165. PML… find
  48. T. Katsch. GateLoop: Fully data-controlled linear recurrence for sequence modeling. ArXiv, 2311.01927, 2023. find
  49. D. Kocetkov, R. Li, L. BenAllal, J. Li, C. Mou, C. Mu nozFerrandis, Y. Jernite, M. Mitchell, S. Hughes, T. Wolf, D. Bahdanau, L. vonWerra, and H. deVries. The Stack: 3 TB of permissively licensed source code. ArXiv, 2211.15533, 2022. find
  50. T. Kohonen. Correlation matrix memories. IEEE Transactions on Computers, C-21(4), 1972. doi: 10.1109/tc.1972.5008975. — [doi:10.1109/tc.1972.5008975](https://doi.org/10.1109/tc.1972.5008975) find
  51. F. Kratzert, D. Klotz, C. Brenner, K. Schulz, and M. Herrnegger. Rainfall-runoff modelling using long short-term memory (LSTM) networks. Hydrology and Earth System Sciences, 22(11):6005–6022, 2018. find
  52. F. Kratzert, D. Klotz, G. Shalev, G. Klambauer, S. Hochreiter, and G. Nearing. Benchmarking a catchment-aware long short-term memory network (LSTM) for large-scale hydrological modeling. ArXiv, 1907.08456, 2019. find
  53. A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, Deptartment of Computer Science, University of Toronto, 2009. find
  54. D. Krotov and J. J. Hopfield. Dense associative memory for pattern recognition. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, pp. 1172–1180. Curran Associates, Inc., 2016. find
  55. D. Krotov and J. J. Hopfield. Dense associative memory is robust to adversarial inputs. ArXiv, 1701.00939, 2017. find
  56. Y. Lakretz, G. Kruszewski, T. Desbordes, D. Hupkes, S. Dehaene, and M. Baroni. The emergence of number and syntax units in LSTM language models. In J. Burstein, C. Doran, and T. Solorio (eds.), Conference of the North American Chapter of the Association for Computational Linguistics, pp. 11–20. Asso…
  57. Y. Li, T. Cai, Y. Zhang, D. Chen, and D. Dey. What makes convolutional models great on long sequence modeling? ArXiv, 2210.09298, 2022. find
  58. P. Liang, R. Bommasani, T. Lee, et al. Holistic evaluation of language models. Annals of the New York Academy of Sciences, 1525:140–146, 2023. find
  59. J. Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, A. Wang, L. Jiang, X. Jia, J. Zhang, J. Zhang, X. Zou, Z. Li, X. Deng, J. Liu, J. Xue, H. Zhou, J. Ma, j. Yu, Y. Li, W. Lin, J. Zhou, J. Tang, and H. Yang. M6: A Chinese multimodal pretrainer. ArXiv, 2103.00823, 2021. find
  60. D. Linsley, J. Kim, V. Veerabadran, C. Windolf, and T. Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018. find
  61. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. URL https://openreview.net/forum?id= Bkg6RiCqY7. ↗
  62. X. Ma, C. Zhou, X. Kong, J. He, L. Gui, G. Neubig, J. May, and L. Zettlemoyer. Mega: Moving average equipped gated attention. ArXiv, 2209.10655, 2022. find
  63. A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Annual Meeting of the Association for Computational Linguistics, volume 49, pp. 142–150, 2011.
  64. I. Magnusson, A. Bhagia, V. Hofmann, et al. Paloma: A benchmark for evaluating language model fit. ArXiv, 2312.10523, 2023.
  65. H. Mehta, A. Gupta, A. Cutkosky, and B. Neyshabur. Long range language modeling via gated state spaces. ArXiv, 2206.13947, 2022. find
  66. S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. In International Conference on Learning Representations (ICRL), 2017. URL https://openreview.net/ forum?id=Byj72udxe. ↗
  67. W. Merrill and A. Sabharwal. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics, 11:531–545, 2023. doi: 10.1162/ tacl_a_00562. find
  68. W. Merrill, J. Petty, and A. Sabharwal. The illusion of state in state-space models. ArXiv, 2404.08819, 2024. find
  69. M. Milakov and N. Gimelshein. Online normalizer calculation for softmax. ArXiv, 1805.02867, 2018.
  70. K. Nakano. Associatron – a model of associative memory. IEEE Transactions on Systems, Man, and Cybernetics, SMC-2(3):380–388, 1972. doi: 10.1109/TSMC.1972.4309133. — [doi:10.1109/tsmc.1972.4309133](https://doi.org/10.1109/tsmc.1972.4309133) find
  71. G. Nearing, D. Cohen, V. Dube, M. Gauch, O. Gilon, S. Harrigan, A. Hassidim, D. Klotz, F. Kratzert, A. Metzger, S. Nevo, F. Pappenberger, C. Prudhomme, G. Shalev, S. Shenzis, T. Y. Tekalign, D. Weitzner, and Y. M. B. Kosko. Global prediction of extreme floods in ungauged watersheds. Nature, 627:559–… — [doi:10.1038/s41586-024-07145-1](https://doi.org/10.1038/s41586-024-07145-1) find
  72. C. Olsson, N. Elhage, N. Nanda, et al. In-context learning and induction heads. ArXiv, 2209.11895, 2022.
  73. A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resurrecting recurrent neural networks for long sequences. In Proceedings of the 40th International Conference on Machine Learning (ICML). JMLR.org, 2023. doi: 10.5555/3618408.3619518.
  74. A. Papasavva, S. Zannettou, E. DeCristofaro, G. Stringhini, and J. Blackburn. Raiders of the lost KeK: 3.5 years of augmented 4chan posts from the politically incorrect board. In International AAAI Conference on Web and Social Media (ICWSM), volume 14, pp. 885–894, 2020. find
  75. D. Paperno, G. Kruszewski, A. Lazaridou, N.-Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, Gemma G. Boleda, and R. Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Annual Meeting of the Association for Computational Linguistics, volume 1, pp. 1525–1534, 2016.
  76. G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei, and J. Launay. The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only. ArXiv, 2306.01116, 2023. find
  77. B. Peng, E. Alcaide, Q. Anthony, et al. RWKV: Reinventing RNNs for the transformer era. ArXiv, 2305.13048, 2023. find
  78. B. Peng, D. Goldstein, Q. Anthony, et al. Eagle and Finch: RWKV with matrix-valued states and dynamic recurrence. ArXiv, 2404.05892, 2024. find
  79. M. Poli, S. Massaroli, E. Nguyen, D. Y. Fu, T. Dao, S. Baccus, Y. Bengio, S. Ermon, and C. Ré. Hyena hierarchy: Towards larger convolutional language models. In Proceedings of the 40th International Conference on Machine Learning (ICML). JMLR.org, 2023. doi: 10.5555/3618408.3619572. find
  80. M. Poli, A. W. Thomas, E. Nguyen, P. Ponnusamy, B. Deiseroth, K. Kersting, T. Suzuki, B. Hie, S. Ermon, C. Ré, C. Zhang, and S. Massaroli. Mechanistic design and scaling of hybrid architectures. ArXiv, 2403.17844, 2024. find
  81. Z. Qin, S. Yang, and Y. Zhong. Hierarchically gated recurrent neural network for sequence modeling. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2023. URL https://openreview.net/forum?id=P1TCHxJwLB. ↗
  82. Z. Qin, S. Yang, W. Sun, X. Shen, D. Li, W. Sun, and Y. Zhong. HGRN2: Gated linear RNNs with state expansion. ArXiv, 2404.07904, 2024. find
  83. D. R. Radev, P. Muthukrishnan, and V. Qazvinian. The ACL anthology network corpus. In Workshop on Text and Citation Analysis for Scholarly Digital Libraries (NLPIR4DL), pp. 54–61. Association for Computational Linguistics, 2009. find
  84. A. Radford, R. Jozefowicz, and I. Sutskever. Learning to generate reviews and discovering sentiment. ArXiv, 1704.01444, 2017.
  85. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. https://openai.com/index/better-language-models, 2019. ↗
  86. J. W. Rae, S. Borgeaud, T. Cai, et al. Scaling language models: Methods, analysis & insights from training Gopher. ArXiv, 2112.11446, 2021. find
  87. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, 1910.10683, 2019.
  88. H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve, V. Greiff, D. Kreil, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. Hopfield networks is all you need. In International Conference on Learning Representations (ICLR). OpenRevie… find
  89. M. Reid, V. Zhong, S. Gururangan, and L. Zettlemoyer. M2D2: A massively multi-domain language modeling dataset. In Conference on Empirical Methods in Natural Language Processing, pp. 964–975, 2022. find
  90. M. Reid, N. Savinov, D. Teplyashin, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv, 2403.05530, 2024.
  91. M. H. Ribeiro, J. Blackburn, B. Bradlyn, E. DeCristofaro, G. Stringhini, S. Long, S. Greenberg, and S. Zannettou. The evolution of the manosphere across the web. In Proceedings of the international AAAI conference on web and social media, volume 15, pp. 196–207, 2021. find
  92. K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. find
  93. T. L. Scao, A. Fan, C. Akiki, et al. BLOOM: A 176B-parameter open-access multilingual language model. ArXiv, 2211.05100, 2022. find
  94. I. Schlag, K. Irie, and J. Schmidhuber. Linear transformers are secretly fast weight programmers. In M. Meila and T. Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 9355–9366. PMLR, 2021.
  95. J. Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural Computation, 4(1):131–139, 1992. find
  96. J. Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015. doi: 10.1016/j.neunet.2014.09.003. find
  97. J. Schulman, B. Zoph, C. Kim, J. Hilton, et al. ChatGPT: Optimizing language models for dialogue. https://openai.com/blog/chatgpt/, 2022. OpenAI Research. ↗
  98. T. J. Sejnowski. Storing covariance with nonlinearly interacting neurons. Journal of Mathematical Biology, 4, 1977. doi: 10.1007/BF00275079. — [doi:10.1007/bf00275079](https://doi.org/10.1007/bf00275079) find
  99. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. ArXiv, 1909.08053, 2019. find
  100. J. T. H. Smith, A. Warrington, and S. W. Linderman. Simplified state space layers for sequence modeling. ArXiv, 2208.04933, 2022. find
  101. D. Soboleva, F. Al-Khateeb, R. Myers, J. R. Steeves, J. Hestness, and N. Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https://www.cerebras.net/blog/ slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023. URL https://huggingface.co/datasets/cerebr… ↗
  102. L. Soldaini, R. Kinney, A. Bhagia, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. ArXiv, 2306.01116, 2023. find
  103. S. Soltan, S. Ananthakrishnan, J. FitzGerald, R. Gupta, W. Hamza, H. Khan, C. Peris, S. Rawls, A. Rosenbaum, A. Rumshisky, C. S. Prakash, M. Sridhar, F. Triefenbach, A. Verma, G. Tur, and P. Natarajan. AlexaTM 20B: Few-shot learning using a large-scale multilingual Seq2Seq model. ArXiv, 2208.01448,…
  104. R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 28. Curran Associates, Inc., 2015. find
  105. Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, J. Xue, J. Wang, and F. Wei. Retentive network: A successor to transformer for large language models. ArXiv, 2307.08621, 2023.
  106. L. Sutawika, L. Gao, H. Schoelkopf, et al. EleutherAI/lm-evaluation-harness: Major refactor, 2023. find
  107. L. Sutawika, H. Schoelkopf, L. Gao, B. Abbasi, S. Biderman, J. Tow, B. fattori, C. Lovering, farzanehnakhaee70, J. Phang, A. Thite, Fazz, T. Wang, N. Muennighoff, Aflah, sdtblck, nopperl, gakada, tttyuntian, researcher2, Chris, J. Etxaniz, H. A. Lee, Z. Kasner, Khalid, J. Hsu, A. Kanekar, P. S. Amma… find
  108. I. Sutskever, O. Vinyals, and Q. V. V. Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27 (NIPS’13), pp. 3104–3112. Curran Associates, Inc., 2014.
  109. Y. Tay, D. Bahri, D. Metzler, D.-C. Juan, Z. Zhao, and C. Zheng. Synthesizer: Rethinking self-attention in transformer models. ArXiv, 2005.00743, 2020. find
  110. Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations (ICRL), 2021. URL https://openreview.net/forum?id=qVyeW-grC2k. ↗
  111. R. Thoppilan, D. deFreitas, J. Hall, et al. LaMDA: Language models for dialog applications. ArXiv, 2201.08239, 2022.
  112. TogetherComputer. Redpajama: an open dataset for training large language models, 2023. URL https://github.com/togethercomputer/RedPajama-Data. ↗
  113. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. ArXiv, 2302.1397, 2023.
  114. D. Vadas and J. R. Curran. Parsing noun phrases in the Penn Treebank. Computational Linguistics, 37(4):753–809, 2011.
  115. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pp. 5998–6008. Curran Associates, Inc., 2017.
  116. O. Vinyals, T. Ewalds, S. Bartunov, et al. Starcraft II: A new challenge for reinforcement learning. ArXiv, 1708.04782, 2017. find
  117. J. Wang, J. N. Yan, A. Gu, and A. M. Rush. Pretraining without attention. ArXiv, 2212.10544, 2022. find
  118. S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma. Linformer: Self-attention with linear complexity. ArXiv, 2006.04768, 2020. find
  119. S. Wang, Y. Sun, Y. Xiang, et al. ERNIE 3.0 Titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation. ArXiv, 2112.12731, 2021.
  120. Y. Wu and K. He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3–19, 2018. find
  121. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Conference of the North American Chapter of the Association for Computational Linguistics, pp. 483–498, 2021. doi: 10.18653/v1/2021.naacl…
  122. S. Yang and Y. Zhang. FLA: A Triton-based library for hardware-efficient implementations of linear attention mechanism, 2024. URL https://github.com/sustcsonglin/ flash-linear-attention. ↗
  123. S. Yang, B. Wang, Y. Shen, R. Panda, and Y. Kim. Gated linear attention transformers with hardware-efficient training. ArXiv, 2312.06635, 2023. find
  124. S. Zannettou, B. Bradlyn, E. DeCristofaro, H. Kwak, M. Sirivianos, G. Stringini, and J. Blackburn. What is Gab: A bastion of free speech or an alt-right echo chamber. In The Web Conference, pp. 1007–1014, 2018. doi: 10.1145/3184558.3191531. — [doi:10.1145/3184558.3191531](https://doi.org/10.1145/3184558.3191531) find
  125. W. Zaremba and I. Sutskever. Learning to execute. ArXiv, 1410.4615, 2014. find
  126. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, 2019. find
  127. A. Zeng, X. Liu, Z. Du, et al. GLM-130B: An open bilingual pre-trained model. ArXiv, 2210.02414, 2022.
  128. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. OPT: Open pre-trained transformer language models. ArXiv, 2205.01068, 2022.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/beck-2024-xlstm-extended-long-short-term/