Elamur Research⌕
☰
home / papers / 2026 / Lightweight LLM Agent Memory with Small Language Models

Lightweight LLM Agent Memory with Small Language Models

Paper · 2026 · Jiaquan Zhang, Chaoning Zhang, Shuxu Chen et al.

Summary

LLM agents need persistent memory to stay consistent across long, multi-turn interactions. Existing approaches force a choice: similarity-search memory is fast but its accuracy varies, while memory managed through repeated large-model calls is more accurate but adds latency at every turn. LightMem addresses this by giving the frequent, structured online memory tasks to small language models (SLMs) and moving heavy consolidation to an offline stage.

Memory is organized into three tiers:

  • STM: the context window.
  • MTM: per-user episodic summaries tagged with user identifiers.
  • LTM: a de-identified, graph-structured knowledge base distilled offline by a large-context model.

Three SLMs handle the online work. A Controller plans hypothetical queries and the retrieval budget, a Selector re-ranks retrieved candidates for semantic consistency, and a Writer updates MTM after each turn.

On LoCoMo and DialSim, the system improves results across model scales while keeping latency and prompt size low. This makes it attractive for deployments where compute is limited.

Key points

  • Two-stage retrieval under a fixed Top-K budget:
    • Metadata-filtered vector search first collects 2K candidates, split evenly across the hypothetical queries.
    • The Selector (SLM-2) then keeps the most relevant half.
  • Accuracy and cost:
    • Average F1 gain of about 2.5 on LoCoMo.
    • Median retrieval latency of 83 ms (P95,167 ms) and median end-to-end latency of 581 ms with GPT-4o-mini.
    • Effective context of about 1K tokens, compared with roughly 16K for MemGPT.
  • Robustness as memory grows (DialSim, Llama-3.2-1B): LightMem's advantage over plain vector retrieval rises from 0.03 F1 at about 100 MTM entries to 0.29 F1 at 10,000 entries.
  • Error injection on DialSim:
    • Removing SLM-2 re-ranking or corrupting half of the writes lowers F1 from 4.12 to 3.83 and 3.78 respectively.
    • Combining all perturbations collapses F1 to 1.85.
  • Update-gap test (LoCoMo multi-hop): querying MTM and LTM together reaches 28.85 F1, compared with 19.45 for LTM only and 20.12 for MTM only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations🎓City University of Hong KongHong Kong · China45 works · 3,014 citations🎓University of Electronic Science and Technology of ChinaChengdu · China31 works · 2,012 citations🎓Kyung Hee UniversitySeoul · South Korea5 works · 176 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Jiaquan Zhang, Chaoning Zhang, Shuxu Chen, Zhenzhen Huang, Pengcheng Zheng, Zhicheng Wang, Ping Guo, Fan Mo, Sung-Ho Bae, Jie Zou, Jiwei Wei and Yang Yang. 2026. Lightweight LLM Agent Memory with Small Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.588

@inproceedings{zhang2026lightweight,
  title = {{Lightweight LLM Agent Memory with Small Language Models}},
  author = {Jiaquan Zhang, Chaoning Zhang, Shuxu Chen, Zhenzhen Huang, Pengcheng Zheng, Zhicheng Wang, Ping Guo, Fan Mo, Sung-Ho Bae, Jie Zou, Jiwei Wei and Yang Yang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.588},
  url = {https://doi.org/10.18653/v1/2026.acl-long.588}
}

References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A Asai, Z Wu, Y Wang, A Sil, and H Self-RAG Hajishirzi. Learning to retrieve, generate, and critique through self-reflection. arxiv 2023. arXiv preprint arXiv:2310.11511.
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  3. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  4. Haixia Han, Jiaqing Liang, Jie Shi, Qianyu He, and Yanghua Xiao. 2024. Small language model can self-correct. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18162– 18170. find
  5. Kostas Hatalis, Despina Christou, Joshua Myers, Steven Jones, Keith Lambert, Adam Amos-Binks, Zohreh Dannenhauer, and Dustin Dannenhauer. 2023. Memory matters: The need to improve long-term memory in llm-agents. In Proceedings of the AAAI Symposium Series, volume 2, pages 277–280. find
  6. Chuanyang Hong and Qingyun He. 2025. Enhancing memory retrieval in generative agents through llm-trained cross attention networks. Frontiers in Psychology, 16:1591618. find
  7. Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. 2025. Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
  8. Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024. Understanding the planning of llm agents: A survey. arXiv preprint arXiv:2402.02716. find
  9. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  10. Jiho Kim, Woosog Chay, Hyeonji Hwang, Daeun Kyung, Hyunseung Chung, Eunbyeol Cho, Yohan Jo, and Edward Choi. 2024. Dialsim: A real-time simulator for evaluating long-term multi-party dialogue understanding of conversation systems. arXiv preprint arXiv:2406.13144. — [arXiv:2406.13144](https://arxiv.org/abs/2406.13144) find
  11. Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John F. Canny, and Ian Fischer. 2024. A human-inspired reading agent with gist memory of very long contexts. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024.
  12. Guangliang Liu, Zhiyu Xue, Xitong Zhang, Rongrong Wang, and Kristen Johnson. 2025. Smaller large language models can do moral self-correction. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pages 56–65. find
  13. Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2023. Teaching small language models to reason. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 2: short papers), pages 1773–1781.
  14. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok… find
  15. Charles Packer, Vivian Fang, Shishir_G Patil, Kevin Lin, Sarah Wooders, and Joseph_E Gonzalez. 2023. Memgpt: Towards llms as operating systems. find
  16. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22. find
  17. Neelabh Sinha, Vinija Jain, and Aman Chadha. 2025. Are small language models ready to compete with large language models for practical applications? In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), pages 365–398. find
  18. Haoran Tan, Zeyu Zhang, Chen Ma, Xu Chen, Quanyu Dai, and Zhenhua Dong. 2025. Membench: Towards more comprehensive evaluation on the memory of llm-based agents. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 19336–19352. Assoc… find
  19. Xudong Wang, Chaoning Zhang, Jiaquan Zhang, Chenghao Li, Qigan Sun, Sung-Ho Bae, Peng Wang, Ning Xie, Jie Zou, Yang Yang, and 1 others. 2026. Efficient and interpretable multi-agent llm routing via ant colony optimization. arXiv preprint arXiv:2603.12933. — [arXiv:2603.12933](https://arxiv.org/abs/2603.12933) find
  20. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025. A-mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110.
  21. Jiaquan Zhang, Qigan Sun, Chaoning Zhang, Xudong Wang, Zhenzhen Huang, Yitian Zhou, Pengcheng Zheng, Chi lok Andy Tai, Sung-Ho Bae, Zeyu Ma, Caiyan Qin, Jinyu Guo, Yang Yang, and Hengtao Shen. 2026a. Tda-rc: Task-driven alignment for knowledge-based reasoning chains in large language models. find
  22. Jiaquan Zhang, Chaoning Zhang, Shuxu Chen, Yibei Liu, Chenghao Li, Qigan Sun, Shuai Yuan, Fach-rina Dewi Puspitasari, Dongshen Han, Guoqing Wang, and 1 others. 2026b. Text summarization via global structure awareness. arXiv preprint arXiv:2602.09821. — [arXiv:2602.09821](https://arxiv.org/abs/2602.09821) find
  23. Jiaquan Zhang, Chaoning Zhang, Shuxu Chen, Xudong Wang, Zhenzhen Huang, Pengcheng Zheng, Shuai Yuan, Sheng Zheng, Qigan Sun, Jie Zou, and 1 others. 2026c. Learning global hypothesis space for enhancing synergistic reasoning chain. arXiv preprint arXiv:2602.09794. — [arXiv:2602.09794](https://arxiv.org/abs/2602.09794) find
  24. Malu Zhang, Wenjie Wei, Zijian Zhou, Wanlong Liu, Jie Zhang, Ammar Belatreche, and Yang Yang. 2025a. Spike-driven lightweight large language model with evolutionary computation. IEEE Transactions on Evolutionary Computation. find
  25. Qinyao Zhang, Bin Guo, Yao Jing, Yan Liu, and Zhiwen Yu. 2024. Mindmemory: Augmented llm with long-term memory and mental personality. In CCF Conference on Computer Supported Cooperative Work and Social Computing, pages 462–476. Springer. find
  26. Zeyu Zhang, Quanyu Dai, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2025b. A survey on the memory mechanism of large language model-based agents. ACM Trans. Inf. Syst., 43(6):155:1–155:47. find
  27. Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731. find
  28. Kunlun Zhu, Hongyi Du, Zhaochen Hong, Xiaocheng Yang, Shuyi Guo, Daisy Zhe Wang, Zhenhailong Wang, Cheng Qian, Robert Tang, Heng Ji, and 1 others. 2025. Multiagentbench: Evaluating the collaboration and competition of llm agents. In Proceedings of the 63rd Annual Meeting of the Association for Compu… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-lightweight-llm-agent-memory-small/