Elamur Research
☰
home / papers / 2019 / Compressive Transformers for Long-Range Sequence Modelling

Compressive Transformers for Long-Range Sequence Modelling

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1911.05507

Summary

Transformer language models keep detailed activations of past tokens, but storing and attending over long histories gets expensive. Models like TransformerXL handle this by deleting old memories once a fixed-size buffer fills. The Compressive Transformer from DeepMind keeps that information in a smaller form instead. Activations about to leave the regular memory are compressed into a second, coarser "compressed memory." For example, a strided 1D convolution can merge every c slots into one. Attention then runs over both memories together. The compression network trains on its own objective, which checks whether attention over the compressed slots reproduces attention over the originals. This lets the model look much further back without heavy extra compute and without custom sparse kernels. The model set new language-modelling records on WikiText-103 and Enwik8. It also modelled raw speech audio and worked as the memory of an IMPALA reinforcement-learning agent. The authors also released PG-19, a large open-vocabulary benchmark built from whole Project Gutenberg books. It is meant to push research on modelling very long contexts.

Key points

  • The model reached 17.1 perplexity on WikiText-103 (word level) and 0.97 bpc on Enwik8 (character level), both state of the art at the time. The biggest gains on WikiText-103 were on rare words.
  • The compression functions tested were max/mean pooling, 1D convolution, dilated convolution and a "most-used" scheme based on attention usage. The best training objective was the lossy attention-reconstruction loss, which beat both the auto-encoding loss and plain backpropagation from the task loss.
  • The maximum temporal range grows to l × (n_m + c·n_cm), while the attention cost only grows with n_m + n_cm.
  • PG-19 is drawn from Project Gutenberg books and is over twice the size of earlier LM benchmarks, with far longer contexts per document.
  • On high-frequency speech it trended toward better likelihood than TransformerXL and WaveNet. It also served as agent memory on an object-matching RL task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

277 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 254 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1911.05507 · arXiv 1911.05507

@misc{rae2019compressive,
  title = {{Compressive Transformers for Long-Range Sequence Modelling}},
  author = {Rae, Jack W. and Potapenko, Anna and Jayakumar, Siddhant M. and Lillicrap, Timothy P.},
  year = {2019},
  doi = {10.48550/arxiv.1911.05507},
  eprint = {1911.05507},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1911.05507}
}

References

48 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Al-Rfou, D. Choe, N. Constant, M. Guo, and L. Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3159–3166, 2019. find
  2. A. Baevski and M. Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853, 2019. find
  3. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. find our card
  4. S. Bai, J. Z. Kolter, and V. Koltun. Convolutional sequence modeling revisited, 2018a. URL https://openreview.net/forum?id=rk8wKk-R-. ↗
  5. S. Bai, J. Z. Kolter, and V. Koltun. Trellis networks for sequence modeling. arXiv preprint arXiv:1810.06682, 2018b. — [arXiv:1810.06682](https://arxiv.org/abs/1810.06682) find
  6. C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik, J. Schrittwieser, K. Anderson, S. York, M. Cant, A. Cain, A. Bolton, S. Gaffney, H. King, D. Hassabis, S. Legg, and S. Petersen. Deepmind lab. CoRR, abs/1612.03801, 2016. URL htt… find
  7. D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993–1022, Mar. 2003. ISSN 1532-4435. find our card
  8. J. Bradbury, S. Merity, C. Xiong, and R. Socher. Quasi-recurrent neural networks. arXiv preprint arXiv:1611.01576, 2016. — [arXiv:1611.01576](https://arxiv.org/abs/1611.01576) find
  9. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013. find our card
  10. R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. find
  11. J. Chung, S. Ahn, and Y. Bengio. Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv:1609.01704, 2016. find
  12. Z. Dai, Z. Yang, Y. Yang, W. W. Cohen, J. Carbonell, Q. V. Le, and R. Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. find
  13. Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier. Language modeling with gated convolutional networks. arXiv preprint arXiv:1612.08083, 2016. find our card
  14. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  15. L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1406–1415, 2018. find
  16. E. Grave, A. Joulin, and N. Usunier. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426, 2016. find
  17. A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. find
  18. A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. find
  19. A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwińska, S. G. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471, 2016. find
  20. D. Ha, A. Dai, and Q. V. Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016. — [arXiv:1609.09106](https://arxiv.org/abs/1609.09106) find
  21. F. Hill, A. Bordes, S. Chopra, and J. Weston. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301, 2015. find our card
  22. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  23. A. Holtzman, J. Buys, M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. find
  24. M. Hutter. The human knowledge compression contest. URL http://prize. hutter1. net, 6, 2012. find
  25. N. Kalchbrenner, L. Espeholt, K. Simonyan, A. v. d. Oord, A. Graves, and K. Kavukcuoglu. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099, 2016. — [arXiv:1610.10099](https://arxiv.org/abs/1610.10099) find
  26. D. P. Kingma and J. Ba. arXiv:1412.6980, 2014. find
  27. Adam: A method for stochastic optimization. T. Kočiskỳ, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328, 2018. find our card
  28. B. Krause, L. Lu, I. Murray, and S. Renals. Multiplicative lstm for sequence modelling. arXiv preprint arXiv:1609.07959, 2016. — [arXiv:1609.07959](https://arxiv.org/abs/1609.07959) find
  29. B. Krause, E. Kahembwe, I. Murray, and S. Renals. Dynamic evaluation of transformer language models. CoRR, abs/1904.08378, 2019. URL http://arxiv.org/abs/1904.08378. — [arXiv:1904.08378](https://arxiv.org/abs/1904.08378) ↗
  30. G. Lample, A. Sablayrolles, M. Ranzato, L. Denoyer, and H. Jégou. Large memory layers with product keys. arXiv preprint arXiv:1907.05242, 2019. — [arXiv:1907.05242](https://arxiv.org/abs/1907.05242) find
  31. S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. find our card
  32. T. Mikolov, M. Karafiát, L. Burget, J. Černockỳ, and S. Khudanpur. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association, 2010. find
  33. A. Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. Driessche, E. Lockhart, L. Cobo, F. Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International Conference on Machine Learning, pages 3915–3923, 2018. find
  34. A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016. find our card
  35. D. Paperno, G. Kruszewski, A. Lazaridou, Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, R. Fernández, K. Erk, et al. The lambada dataset: Word prediction requiring a broad discourse context. Association for Computational Linguistics, 2016. find our card
  36. J. Rae, J. J. Hunt, I. Danihelka, T. Harley, A. W. Senior, G. Wayne, A. Graves, and T. Lillicrap. Scaling memory-augmented neural networks with sparse reads and writes. In Advances in Neural Information Processing Systems, pages 3621–3629, 2016. find
  37. J. W. Rae, C. Dyer, P. Dayan, and T. P. Lillicrap. Fast parametric learning with activation memorization. arXiv preprint arXiv:1803.10049, 2018. — [arXiv:1803.10049](https://arxiv.org/abs/1803.10049) find
  38. B. A. Richards and P. W. Frankland. The persistence and transience of memory. Neuron, 94(6): 1071–1084, 2017. find
  39. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. Nature, 323(6088):533, 1986. find
  40. A. Santoro, R. Faulkner, D. Raposo, J. Rae, M. Chrzanowski, T. Weber, D. Wierstra, O. Vinyals, R. Pascanu, and T. Lillicrap. Relational recurrent neural networks. In Advances in Neural Information Processing Systems, pages 7299–7310, 2018. find
  41. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019. find
  42. S. Smith, P. jan Kindermans, C. Ying, and Q. V. Le. Don’t decay the learning rate, increase the batch size. 2018. URL https://openreview.net/pdf?id=B1Yy1BxCZ. arXiv preprint S. Sukhbaatar, E. Grave, P. Bojanowski, and A. Joulin. Adaptive attention span in transformers. arXiv preprint arXiv:1905.0779… ↗
  43. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017. find
  44. F. Wu, A. Fan, A. Baevski, Y. N. Dauphin, and M. Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.10430, 2019. find our card
  45. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019. find our card
  46. L. Zhou, Y. Zhou, J. J. Corso, R. Socher, and C. Xiong. End-to-end dense video captioning with masked transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8739–8748, 2018. find
  47. Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27, 2015. find our card
  48. J. G. Zilly, R. K. Srivastava, J. Koutnı́k, and J. Schmidhuber. Recurrent highway networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 4189– 4198. JMLR. org, 2017. find