Elamur Research⌕
☰
home / papers / 2021 / Linear Transformers Are Secretly Fast Weight Programmers

Linear Transformers Are Secretly Fast Weight Programmers

Paper · 2021 · Imanol Schlag, Kazuki Irie and Jurgen Schmidhuber

Summary

The paper shows that linear Transformers, which replace softmax attention with a kernel-based approximation, are mathematically the same as the outer-product Fast Weight Programmers (FWPs) Schmidhuber proposed in the early 1990s, apart from a normalisation term. In an FWP, a slow network trained by gradient descent writes key–value associations into the fast weights of another network. Seen this way, linear attention is a finite associative memory. Once a sequence holds more items than the key space has orthogonal directions (d_dot), retrieval starts mixing stored values.

To address this, the authors swap the purely additive write for a delta-rule-style update. This lets the model overwrite outdated associations, and it learns its own per-step learning rates. They also introduce DPFP, a simple new kernel for linearising attention. They test these ideas on synthetic retrieval, WMT14 English–German translation and WikiText-103 language modelling.

The delta-rule model (Delta Net) remains stable on unbounded context, where the standard linear Transformer fails. This matters for constant-memory sequence models.

Key points

  • Formal link: linearised self-attention can be written as fast-weight updates, W ← W + v ⊗ φ(k), with normalisation. Storing more than d_dot associations causes retrieval errors.
  • On WikiText-103 (medium config, limited context), the best Delta Net setting omits both positional encoding and extra attention normalisation. It reaches 28.1 valid and 31.1 test perplexity.
  • Without truncating context, the Delta Net scores 27.8 valid and 29.4 test perplexity with a 0.13M state. The plain Linear Transformer exceeds 260, and Transformer-XL at the same state size gets about 65.5 test.
  • Transformer-XL is still better with large states, for example 25.5 test at a 6.29M state.
  • The delta update adds little cost: 63K vs 66K words/sec and 14 vs 13 GB memory. A regular PyTorch Transformer runs at 33K words/sec and uses 17 GB. Code is available at github.com/ischlag/fast-weight-transformers (ICML 2021).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Università della Svizzera italianaLugano · Switzerland10 works · 8,004 citations🔬Dalle Molle Institute for Artificial Intelligence ResearchLugano · Switzerland10 works · 8,825 citations🎓University of Applied Sciences and Arts of Southern SwitzerlandManno · Switzerland8 works · 7,374 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Imanol Schlag, Kazuki Irie and Jurgen Schmidhuber. 2021. Linear Transformers Are Secretly Fast Weight Programmers. arXiv:2102.11174.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2102.11174 · arXiv 2102.11174

@misc{schlag2021linear,
  title = {{Linear Transformers Are Secretly Fast Weight Programmers}},
  author = {Imanol Schlag, Kazuki Irie and Jurgen Schmidhuber},
  year = {2021},
  doi = {10.48550/arxiv.2102.11174},
  eprint = {2102.11174},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2102.11174}
}

References

70 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Al-Rfou, R., Choe, D., Constant, N., Guo, M., and Jones, L. Character-level language modeling with deeper self-attention. In Proc. Conference on Artificial Intelligence (AAAI), pp. 3159–3166, Honolulu, HI, USA, January 2019.
  2. Ba, J., Hinton, G. E., Mnih, V., Leibo, J. Z., and Ionescu, C. Using fast weights to attend to the recent past. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 4331–4339, Barcelona, Spain, December 2016. find
  3. Baevski, A. and Auli, M. Adaptive input representations for neural language modeling. In Int. Conf. on Learning Representations (ICLR), New Orleans, LA, USA, May 2019.
  4. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Int. Conf. on Learning Representations (ICLR), San Diego, CA, USA, May 2015.
  5. Bello, I. Lambdanetworks: Modeling long-range interactions without attention. In Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021. find
  6. Cheng, J., Dong, L., and Lapata, M. Long short-term memory-networks for machine reading. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pp. 551–561, Austin, TX, USA, November 2016.
  7. Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. Rethinking attention with performers. In Int. Conf. on Learning Representations (ICLR), Virtual only, 2021.
  8. Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). In Int. Conf. on Learning Representations (ICLR), San Juan, Puerto Rico, May 2016.
  9. Dai, Z., Yang, Z., Yang, Y., Cohen, W. W., Carbonell, J., Le, Q. V., and Salakhutdinov, R. Transformer-XL: Attentive language models beyond a fixed-length context. In Proc. Association for Computational Linguistics (ACL), pp. 2978–2988, Florence, Italy, July 2019. find
  10. Demircigil, M., Heusel, J., Löwe, M., Upgang, S., and Vermet, F. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168(2): 288–299, 2017. find
  11. Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. North American Chapter of the Association for Computational Linguistics on Human Language Technologies (NAACL-HLT), pp. 4171– 4186, Minneapolis, MN, USA, June…
  12. Feldman, J. A. Dynamic connections in neural networks. Biological cybernetics, 46(1):27–39, 1982. find
  13. Galanti, T. and Wolf, L. On the modularity of hypernetworks. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only, 2020. find
  14. Graves, A. Generating sequences with recurrent neural networks. Preprint arXiv:1308.0850, 2013. find
  15. Greff, K., van Steenkiste, S., and Schmidhuber, J. On the binding problem in artificial neural networks. Preprint arXiv:2012.05208, 2020. — [arXiv:2012.05208](https://arxiv.org/abs/2012.05208) find
  16. Ha, D., Dai, A., and Le, Q. V. Hypernetworks. In Int. Conf. on Learning Representations (ICLR), Toulon, France, April 2017. find
  17. Hanson, S. J. A stochastic version of the delta rule. Physica D: Nonlinear Phenomena, 42(1-3):265–272, 1990. find
  18. Hebb, D. O. The organization of behavior: a neuropsy-cholocigal theory. A Wiley Book in Clinical Psychology, 62:78, 1949. find
  19. Hinton, G. E. and Plaut, D. C. Using fast weights to deblur old memories. In Proc. Conf. of Cognitive Science Society, pp. 177–186, Seatle, WA, USA, July 1987. find
  20. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  21. Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proc. of the national academy of sciences, 79(8):2554–2558, 1982. find
  22. Irie, K., Zeyer, A., Schlüter, R., and Ney, H. Language modeling with deep Transformers. In Proc. Interspeech, pp. 3905–3909, Graz, Austria, September 2019a. find
  23. Irie, K., Zeyer, A., Schlüter, R., and Ney, H. Training language models for long-span cross-sentence evaluation. In Proc. IEEE Automatic Speech Recog. and Understanding Workshop (ASRU), pp. 419–426, Sentosa, Singapore, December 2019b. find
  24. Irie, K., Gerstenberger, A., Schlüter, R., and Ney, H. How much self-attention do we need? Trading attention for feed-forward layers. In Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), pp. 6154– 6158, Virtual only, May 2020. find
  25. Jia, X., De Brabandere, B., Tuytelaars, T., and Gool, L. V. Dynamic filter networks. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 667–675, Barcelona, Spain, 2016. find
  26. Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proc. Int. Conf. on Machine Learning (ICML), Virtual only, July 2020. find
  27. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. Preprint arXiv:1412.6980, 2014.
  28. Kirsch, L. and Schmidhuber, J. Meta learning backpropagation and improving it. NeurIPS Workshop on Meta-Learning, 2020. find
  29. Klein, B., Wolf, L., and Afek, Y. A dynamic convolutional layer for short rangeweather prediction. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 4840–4848, Boston, MA, USA, June 2015. find
  30. Kohonen, T. Correlation matrix memories. IEEE Transactions on Computers, 21(4):353–359, 1972. find
  31. Kosko, B. Bidirectional associative memories. IEEE Transactions on Systems, Man, and Cybernetics, 18(1):49–60, 1988. find
  32. Krotov, D. and Hopfield, J. J. Dense associative memory for pattern recognition. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 1172–1180, Barcelona, Spain, December 2016. find
  33. Lin, Z., Feng, M., Santos, C. N. d., Yu, M., Xiang, B., Zhou, B., and Bengio, Y. A structured self-attentive sentence embedding. In Int. Conf. on Learning Representations (ICLR), Toulon, France, April 2017.
  34. Little, W. A. The existence of persistent states in the brain. Mathematical biosciences, 19(1-2):101–120, 1974. find
  35. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In Int. Conf. on Learning Representations (ICLR), Toulon, France, April 2017.
  36. Miconi, T., Stanley, K., and Clune, J. Differentiable plasticity: training plastic neural networks with backpropagation. In Proc. Int. Conf. on Machine Learning (ICML), pp. 3559–3568, Stockholm, Sweden, July 2018. find
  37. Miconi, T., Rawal, A., Clune, J., and Stanley, K. O. Back-propamine: training self-modifying neural networks with differentiable neuromodulated plasticity. In Int. Conf. on Learning Representations (ICLR), New Orleans, LA, USA, May 2019. find
  38. Noh, H., Seo, P. H., and Han, B. Image question answering using convolutional neural network with dynamic parameter prediction. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 30–38, Las Vegas, NV, USA, 2016. find
  39. Ott, M., Edunov, S., Grangier, D., and Auli, M. Scaling neural machine translation. In Proc. Conf. on Machine Translation: Research Papers, pp. 1–9, Brussels, Belgium, October 2018.
  40. Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. North American Chapter of the Association for Computational Linguistics on Human Language Technologies (NAACL-HLT), Demonstrations, pp. 48–53,…
  41. Palm, G. On associative memory. Biological cybernetics, 36(1):19–31, 1980. find
  42. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. Association for Computational Linguistics (ACL), pp. 311–318, Philadelphia, PA, USA, July 2002.
  43. Parikh, A. P., Täckström, O., Das, D., and Uszkoreit, J. A decomposable attention model for natural language inference. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pp. 2249–2255, Austin, TX, USA, November 2016.
  44. Paszke, A. et al. Pytorch: An imperative style, high-performance deep learning library. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 8026–8037, Vancouver, Canada, December 2019.
  45. Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N. A., and Kong, L. Random feature attention. In Int. Conf. on Learning Representations (ICLR), Virtual only, 2021. find
  46. Munkhdalai, T. and Trischler, A. Metalearning with hebbian fast weights. Preprint arXiv:1807.05076, 2018. — [arXiv:1807.05076](https://arxiv.org/abs/1807.05076) find
  47. Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. FiLM: Visual reasoning with a general conditioning layer. In Proc. AAAI Conf. on Artificial Intelligence, pp. 3942–3951, New Orleans, LA, USA, February 2018.
  48. Munkhdalai, T. and Yu, H. Meta networks. In Proc. Int. Conf. on Machine Learning (ICML), pp. 2554–2563, Sydney, Australia, August 2017. find
  49. Post, M. A call for clarity in reporting BLEU scores. In Proc. Conf. on Machine Translation, pp. 186–191, Brussels, Belgium, October 2018.
  50. Munkhdalai, T., Sordoni, A., Wang, T., and Trischler, A. Metalearned neural memory. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pp. 13310– 13321, Vancouver, Canada, December 2019. find
  51. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multi-task learners. [Online]. : https://blog.openai.com/better-language-models/, 2019. ↗
  52. Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. In Int. Conf. on Learning Representations (ICLR), Virtual only, April 2020.
  53. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014.
  54. Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Adler, T., Kreil, D., Kopp, M. K., Klambauer, G., Brandstetter, J., and Hochreiter, S. Hopfield networks is all you need. In Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021. find
  55. Steinbuch, K. Die lernmatrix. Kybernetik, 1(1):36–45, 1961. find
  56. Schlag, I. and Schmidhuber, J. Gated fast weights for onthe-fly neural program generation. In NIPS Metalearning Workshop, Long Beach, CA, USA, December 2017. find
  57. Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. Long range arena: A benchmark for efficient transformers. In Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021.
  58. Schlag, I. and Schmidhuber, J. Learning to reason with third order tensor products. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 9981–9993, Montréal, Canada, December 2018. find
  59. Schlag, I., Munkhdalai, T., and Schmidhuber, J. Learning associative inference using fast weight memory. In Int. Conf. on Learning Representations (ICLR), Virtual only, May 2021. find
  60. Schmidhuber, J. Learning to control fast-weight memories: An alternative to recurrent nets. Technical Report FKI-147-91, Institut für Informatik, Technische Universität München, March 1991. find
  61. Schmidhuber, J. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131–139, 1992. find
  62. Schmidhuber, J. Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets. In International Conference on Artificial Neural Networks (ICANN), pp. 460–463, Amsterdam, Netherlands, September 1993. find
  63. Schmidhuber, J. 26 March 1991: Neural nets learn to program neural nets with fast weights—like today’s Transformer variants. 2021: New stuff!, AI Blog, 2021. URL https://people.idsia.ch/˜juergen/ fast-weight-programmer-1991-transformer. html. ↗
  64. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Proc. Association for Computational Linguistics (ACL), pp. 1715– 1725, Berlin, Germany, August 2016.
  65. Shen, Z., Zhang, M., Zhao, H., Yi, S., and Li, H. Efficient attention: Attention with linear complexities. Preprint arXiv:1812.01243, 2018. — [arXiv:1812.01243](https://arxiv.org/abs/1812.01243) find
  66. Smolensky, P. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):159–216, 1990. find
  67. Steinbuch, K. and Piske, U. A. W. Learning matrices and their applications. IEEE Transactions on Electronic Computers, 12(6):846–862, 1963. find
  68. Tsai, Y.-H. H., Bai, S., Yamada, M., Morency, L.-P., and Salakhutdinov, R. Transformer dissection: An unified understanding for transformer’s attention via the lens of kernel. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pp. 4344–4353, Hong Kong, China, November 2019. find
  69. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NIPS), pp. 5998–6008, Long Beach, CA, USA, December 2017. von der Malsburg, C. The correlation theory of…
  70. Widrow, B. and Hoff, M. E. Adaptive switching circuits. In Proc. IRE WESCON Convention Record, pp. 96–104, Los Angeles, CA, USA, August 1960. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schlag-2021-linear-transformers-secretly-fast-weight/