Elamur Research⌕
☰
home / papers / 2023 / Language Modeling Is Compression

Language Modeling Is Compression

Summary

The paper revisits the old equivalence between prediction and lossless compression and uses it to evaluate large pretrained models. Any model that assigns probabilities to the next symbol can drive an arithmetic coder, so a lower log-loss means a shorter code. The authors, from Google DeepMind and Meta AI/Inria, test pretrained models in an offline setting. Parameters stay frozen and each data chunk is compressed using only the model's context window. They compare Chinchilla models (1B, 7B, 70B) and Transformers trained on enwik against standard compressors on text, image and audio data. Despite being trained mostly on text, the largest model beats domain-specific tools such as PNG and FLAC, if model size is ignored. Once the parameters are counted as part of the output, scaling has a limit: beyond some point, bigger models compress a given dataset worse. The authors also show that tokenization does not generally help compression. Finally, they run the equivalence in reverse to sample from gzip as a generative model. The work matters because it offers compression as a principled lens on generalization, scaling and in-context learning.

Key points

  • Chinchilla 70B compresses ImageNet patches to 43.4% of their raw size, compared with 58.5% for PNG. It compresses LibriSpeech samples to 16.4%, compared with 30.3% for FLAC.
  • Inputs are split into 2048-byte chunks. Because the large models expose only their top-k next-token log-probabilities (k = 100), these are renormalized before being passed to the arithmetic coder.
  • The authors propose an adjusted compression rate that includes model size. Under this measure, dataset size puts a hard ceiling on useful model size, so model scaling is not a cure-all.
  • Tokenization acts as a form of pre-compression. It does not usually improve compression rates, but it lets more information fit into the context, which helps prediction.
  • In generation experiments conditioned on 1948 bytes of enwik9 text, Chinchilla samples are coherent while gzip samples are mostly noise. Similar comparisons are shown for image and LibriSpeech audio data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Language models & pretraining · Efficient models & compression

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.10668 · arXiv 2309.10668

@misc{deletang2023language,
  title = {{Language Modeling Is Compression}},
  author = {Deletang, Gregoire and Ruoss, Anian and Duquenne, Paul-Ambroise and Catt, Elliot and Genewein, Tim and Mattern, Christopher and Grau-Moya, Jordi and Wenliang, Li Kevin and Aitchison, Matthew and Orseau, Laurent and Hutter, Marcus and Veness, Joel},
  year = {2023},
  doi = {10.48550/arxiv.2309.10668},
  eprint = {2309.10668},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.10668}
}

References

82 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Benjamin Lukas Cajus Barzen, Fedor Glazov, Jonas Geistert, and Thomas Sikora. Accelerated deep lossless image coding with unified paralleleized GPU coding architecture. In PCS, 2022. find
  2. Fabrice Bellard. Lossless data compression with neural networks. Technical report, Amarisoft, 2019. find
  3. Fabrice Bellard. NNCP v2: Lossless data compression with transformer. Technical report, Amarisoft, 2021. find
  4. Léonard Blier and Yann Ollivier. The description length of deep learning models. In NeurIPS, 2018. find
  5. Rishi Bommasani et al. On the opportunities and risks of foundation models. arXiv:2108.07258, 2021. find our card
  6. Thomas Boutell. PNG (portable network graphics) specification version 1.0. RFC, 1997. find
  7. Tom B. Brown, Benjamin Mannand Nick Ryder, Melanie Subbiah, et al. Language models are few-shot learners. In NeurIPS, 2020. find our card
  8. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv:2303.127… find our card
  9. Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. Scaling transformer to 1m tokens and beyond with RMT. arXiv:2304.11062, 2023. — [arXiv:2304.11062](https://arxiv.org/abs/2304.11062) find
  10. Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv:1710.09282, 2017. — [arXiv:1710.09282](https://arxiv.org/abs/1710.09282) find
  11. John G. Cleary and Ian H. Witten. Data compression using adaptive coding and partial string matching. IEEE Trans. Commun., 1984. find
  12. Josh Coalson. Free lossless audio codec, 2008. URL https://xiph.org/flac . ↗
  13. David Cox. Syntactically informed text compression with recurrent neural networks. arXiv:1608.02893, 2016. — [arXiv:1608.02893](https://arxiv.org/abs/1608.02893) find
  14. Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A. Ortega. Neural networks and the chomsky hierarchy. In ICLR, 2023. find our card
  15. Peter Deutsch. GZIP file format specification version 4.3. RFC, 1996. find
  16. Jarek Duda. Asymmetric numeral systems. arXiv:0902.0271, 2009. — [arXiv:0902.0271](https://arxiv.org/abs/0902.0271) find
  17. Eibe Frank, Chang Chui, and Ian H. Witten. Text categorization using compression models. In Data Compression Conference, 2000. find
  18. Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. arXiv:2307.06945, 2023. find
  19. Tim Genewein, Grégoire Delétang, Anian Ruoss, Li Kevin Wenliang, Elliot Catt, Vincent Dutordoir, Jordi Grau-Moya, Laurent Orseau, Marcus Hutter, and Joel Veness. Memory-based meta-learning on non-stationary distributions. In ICML, 2023. find
  20. Mohit Goyal, Kedar Tatwawadi, Shubham Chandak, and Idoia Ochoa. Deepzip: Lossless data compression using recurrent neural networks. In DCC, 2019. find
  21. Mohit Goyal, Kedar Tatwawadi, Shubham Chandak, and Idoia Ochoa. Dzip: Improved general-purpose lossless compression based on novel neural network modeling. In DCC, 2020. find
  22. Mandy Guo, Joshua Ainslie, David C. Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. Longt5: Efficient text-to-text transformer for long sequences. In NAACL-HLT (Findings), 2022. find our card
  23. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. Training compute-optimal large language models. arXiv:2203.15556, 2022. find our card
  24. Emiel Hoogeboom, Jorn W. T. Peters, Rianne van den Berg, and Max Welling. Integer discrete flows and lossless compression. In NeurIPS, 2019. find
  25. Paul G. Howard and Jeffrey Scott Vitter. Analysis of arithmetic coding for data compression. In Data Compression Conference, 1991. find
  26. David A. Huffman. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 1952. find
  27. Marcus Hutter. Universal Artificial Intellegence - Sequential Decisions Based on Algorithmic Probability. Springer, 2005. find
  28. Marcus Hutter. 500’000C prize for compressing human knowledge, 2006. URL http://prize.hutter1.net . ↗
  29. Zhiying Jiang, Yiqin Dai, Ji Xin, Ming Li, and Jimmy Lin. Few-shot non-parametric learning with deep latent variable model. In NeurIPS, 2022. find
  30. Zhiying Jiang, Matthew Y. R. Yang, Mikhail Tsirlin, Raphael Tang, Yiqin Dai, and Jimmy Lin. "low-resource" text classification: A parameter-free classification method with compressors. In ACL (Findings), 2023. find
  31. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.08361, 2020. find our card
  32. Friso H. Kingma, Pieter Abbeel, and Jonathan Ho. Bit-swap: Recursive bits-back coding for lossless compression with hierarchical latent variables. In ICML, 2019. find
  33. Byron Knoll. CMIX, 2014. URL http://www.byronknoll.com/cmix.html . ↗
  34. Byron Knoll and Nando de Freitas. A machine learning perspective on predictive coding with PAQ8. In DCC, 2012. find
  35. Andrei N. Kolmogorov. On tables of random numbers. Theoretical Computer Science, 1998. find
  36. Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In ACL (1), 2018. find our card
  37. Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP (Demonstration), 2018. find our card
  38. Michael Laskin, Luyu Wang, et al. In-context reinforcement learning with algorithm distillation. In ICLR. OpenReview.net, 2023. find
  39. Ming Li and Paul M. B. Vitányi. An Introduction to Kolmogorov Complexity and Its Applications, 4th Edition. Springer, 2019. find
  40. Qian Liu, Yiling Xu, and Zhu Li. DecMac: A deep context model for high efficiency arithmetic coding. In ICAIIC, 2019. find
  41. David J. C. MacKay. Information theory, inference, and learning algorithms. Cambridge University Press, 2003. find
  42. Matthew V. Mahoney. Fast text compression with neural networks. In FLAIRS, 2000. find
  43. Yu Mao, Yufei Cui, Tei-Wei Kuo, and Chun Jason Xue. TRACE: A fast transformer-based general-purpose lossless compressor. In WWW, 2022. find
  44. Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. Practical full resolution learned lossless image compression. In CVPR, 2019. find
  45. Fabian Mentzer, Luc Van Gool, and Michael Tschannen. Learning better lossless compression using lossy compression. In CVPR, 2020. find
  46. Tomas Mikolov. Statistical Language Models Based on Neural Networks. PhD thesis, Brno Universtiy of Technology, 2012. find
  47. Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. arXiv:2307.04721, 2023. find
  48. Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In NIPS, 2017. find
  49. Juri Opitz. Gzip versus bag-of-words for text classification, 2023. find
  50. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In ICASSP, 2015. find
  51. Richard C. Pasco. Source coding algorithms for fast data compression (ph.d. thesis abstr.). IEEE Trans. Inf. Theory, 1977. find
  52. Igor Pavlov. 7z Format, 2019. URL http://www.7-zip.org/7z.html . ↗
  53. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. Bpe-dropout: Simple and effective subword regularization. In ACL, 2020. find our card
  54. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019. find
  55. Jack W. Rae et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv:2112.11446, 2021. find
  56. Samuel Rathmanner and Marcus Hutter. A philosophical treatise of universal induction. Entropy, 2011. find
  57. Hochang Rhee, Yeong Il Jang, Seyun Kim, and Nam Ik Cho. LC-FDNet: Learned lossless image compression with frequency decomposition network. In CVPR, 2022. find
  58. Jorma Rissanen. Generalized kraft inequality and arithmetic coding. IBM J. Res. Dev., 1976. find
  59. Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. In ACL (2), 2023. find
  60. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 2015. find our card
  61. Ionut Schiopu and Adrian Munteanu. Deep-learning-based lossless image coding. IEEE Trans. Circuits Syst. Video Technol., 2020. find
  62. Ionut Schiopu, Yu Liu, and Adrian Munteanu. CNN-based prediction for lossless coding of photographic images. In PCS, 2018. find
  63. Jürgen Schmidhuber and Stefan Heil. Predictive coding with neural nets: Application to text compression. In NIPS, pp. 1047–1054. MIT Press, 1994. find
  64. Jürgen Schmidhuber and Stefan Heil. Sequential neural text compression. IEEE Trans. Neural Networks, 1996. find
  65. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL (1), 2016. find our card
  66. Claude E. Shannon. A mathematical theory of communication. Bell Syst. Tech. J., 1948. find
  67. Ray J. Solomonoff. A formal theory of inductive inference. part I. Inf. Control., 1964a. find
  68. Ray J. Solomonoff. A formal theory of inductive inference. part II. Inf. Control., 1964b. find
  69. Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantization. In ACL (1), 2022. find
  70. William J. Teahan and David J. Harper. Using Compression-Based Language Models for Text Categorization, pp. 141–165. Springer Netherlands, 2003. find
  71. Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation language models. arXiv:2302.13971, 2023a. find our card
  72. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  73. James Townsend, Thomas Bird, and David Barber. Practical lossless compression with latent variables using bits back coding. In ICLR (Poster), 2019. find
  74. Chandra Shekhara Kaushik Valmeekam, Krishna Narayanan, Dileep Kalathil, Jean-François Cham-berland, and Srinivas Shakkottai. Llmzip: Lossless text compression using large language models. arXiv:2306.04050, 2023. — [arXiv:2306.04050](https://arxiv.org/abs/2306.04050) find
  75. Aäron van den Oord and Benjamin Schrauwen. The student-t mixture as a natural image patch prior with application to image compression. J. Mach. Learn. Res., 2014. find
  76. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. find our card
  77. Joel Veness, Marc G. Bellemare, Marcus Hutter, Alvin Chua, and Guillaume Desjardins. Compress and control. In AAAI, 2015. find
  78. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. find our card
  79. Terry A. Welch. A technique for high-performance data compression. Computer, 1984. find
  80. Frans M. J. Willems, Yuri M. Shtarkov, and Tjalling J. Tjalkens. The context-tree weighting method: basic properties. IEEE Trans. Inf. Theory, 1995. find
  81. Ian H. Witten, Radford M. Neal, and John G. Cleary. Arithmetic coding for data compression. Commun. ACM, 1987. find
  82. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In NeurIPS, 2020. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/deletang-2023-language-modeling-compression/