Elamur Research⌕
☰
home / papers / 2024 / MEMORYLLM: Towards Self-Updatable Large Language Models

MEMORYLLM: Towards Self-Updatable Large Language Models

Paper · 2024 · Yu Wang, Yifan Gao, Xiusi Chen et al.

Summary

Large language models are usually frozen once deployed, which makes adding fresh knowledge difficult. Current workarounds each have drawbacks. Retrieval stores keep growing, fact-level editing struggles with longer passages, and long-context approaches eventually run out of context length. This paper proposes MEMORYLLM, which splits the model into two parts. One is a frozen backbone (Llama2). The other is a constant-size pool of latent "memory tokens" placed in every transformer layer, which serves as the updatable parameters. When the model reads a new text passage, the backbone itself compresses it into a few new memory tokens per layer. An equal number of existing tokens is discarded at random, so older content fades roughly exponentially and the pool never grows. Updates require no back-propagation. The authors report:

  • gains over model-editing and long-context baselines,
  • strong recall in their own knowledge-retention experiments,
  • stable behaviour after close to one million updates.

The work offers a bounded-memory way to keep deployed LLMs current without retraining or external knowledge bases.

Key points

  • Architecture: a Llama2 7B backbone (parameters ϕ, kept fixed) is paired with a memory pool of about 1B parameters (θ). The pool consists of N memory tokens of the embedding dimension d in each layer.
  • Generation: every input token attends to the full memory pool, so cost grows linearly with pool size.
  • Self-update: the last K memory tokens are combined with the new passage's hidden states. The last K outputs become the new memory, and K randomly chosen old tokens are dropped. Knowledge therefore decays at rate K/N, a design inspired by the Ebbinghaus forgetting curve.
  • Evaluation: model-editing benchmarks (compared against MEND, ROME and IKE), long-context QA benchmarks, custom retention tests, and a robustness run of almost a million updates with no observed degradation.
  • Availability: code and model are open-sourced at github.com/wangyu-ustc/MemoryLLM (ICML 2024).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, Los AngelesLos Angeles · United States87 works · 54,961 citations🎓University of California San DiegoSan Diego · United States73 works · 51,948 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang and Julian McAuley. 2024. MEMORYLLM: Towards Self-Updatable Large Language Models. arXiv:2402.04624.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.04624 · arXiv 2402.04624

@misc{wang2024memoryllm,
  title = {{MEMORYLLM: Towards Self-Updatable Large Language Models}},
  author = {Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang and Julian McAuley},
  year = {2024},
  doi = {10.48550/arxiv.2402.04624},
  eprint = {2402.04624},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.04624}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bulatov, A., Kuratov, Y., and Burtsev, M. S. Recurrent memory transformer. In NeurIPS, 2022. find
  2. Burtsev, M. S. and Sapunov, G. V. Memory transformer. CoRR, abs/2006.11527, 2020. URL https://arxiv.org/abs/2006.11527. — [arXiv:2006.11527](https://arxiv.org/abs/2006.11527) ↗
  3. Cao, N. D., Aziz, W., and Titov, I. Editing factual knowledge in language models. In EMNLP (1), pp. 6491–6506. Association for Computational Linguistics, 2021. find
  4. Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022. find
  5. Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, C. D. Fast model editing at scale. In ICLR. OpenReview.net, 2022. find
  6. Chen, S., Wong, S., Chen, L., and Tian, Y. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023a. find
  7. Modarressi, A., Imani, A., Fayyaz, M., and Schütze, H. Ret-llm: Towards a general read-write memory for large language models. arXiv preprint arXiv:2305.14322, 2023. — [arXiv:2305.14322](https://arxiv.org/abs/2305.14322) find
  8. Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307, 2023b.
  9. Moro, G., Ragazzi, L., Valgimigli, L., Frisoni, G., Sartori, C., and Marfia, G. Efficient memory-enhanced transformer for long-document summarization in low-resource regimes. Sensors, 23(7):3542, 2023. find
  10. Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. CoRR, abs/1904.10509, 2019. URL http://arxiv.org/ abs/1904.10509. ↗
  11. Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. find
  12. Computer, T. Redpajama: an open dataset for training large language models, 2023. URL https://github.com/togethercomputer/RedPajama-Data. ↗
  13. Cornia, M., Stefanini, M., Baraldi, L., and Cucchiara, R. Meshed-memory transformer for image captioning. In CVPR, pp. 10575–10584. Computer Vision Foundation / IEEE, 2020. find
  14. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020.
  15. Sukhbaatar, S., Weston, J., Fergus, R., et al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015. find
  16. Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., and Wei, F. A length-extrapolatable transformer. In ACL (1), pp. 14590–14604. Association for Computational Linguistics, 2023.
  17. Zhong, W., Guo, L., Gao, Q., and Wang, Y. Memorybank: Enhancing large language models with long-term memory. arXiv preprint arXiv:2305.10250, 2023. find
  18. Zhu, C., Rawat, A. S., Zaheer, M., Bhojanapalli, S., Li, D., Yu, F. X., and Kumar, S. Modifying memories in transformer models. CoRR, abs/2012.00363, 2020. find
  19. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Ho… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  20. Tworkowski, S., Staniszewski, K., Pacek, M., Wu, Y., Michalewski, H., and Miłoś, P. Focused transformer: Contrastive training for context scaling. arXiv preprint arXiv:2307.03170, 2023. — [arXiv:2307.03170](https://arxiv.org/abs/2307.03170) find
  21. Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. find
  22. Wang, W., Dong, L., Cheng, H., Liu, X., Yan, X., Gao, J., and Wei, F. Augmenting language models with long-term memory. arXiv preprint arXiv:2306.07174, 2023. find
  23. Weston, J., Chopra, S., and Bordes, A. Memory networks. arXiv preprint arXiv:1410.3916, 2014. find
  24. Wu, Q., Lan, Z., Qian, K., Gu, J., Geramifard, A., and Yu, Z. Memformer: A memory-augmented transformer for sequence modeling. In AACL/IJCNLP (Findings), pp. 308–318. Association for Computational Linguistics, 2022. find
  25. Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039, 2023.
  26. Yao, Y., Wang, P., Tian, B., Cheng, S., Li, Z., Deng, S., Chen, H., and Zhang, N. Editing large language models: Problems, methods, and opportunities. CoRR, abs/2305.13172, 2023. find
  27. Zheng, C., Li, L., Dong, Q., Fan, Y., Wu, Z., Xu, J., and Chang, B. Can we edit factual knowledge by in-context learning? arXiv preprint arXiv:2305.12740, 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2024-memoryllm-towards-self-updatable-large/