Elamur Research⌕
☰
home / papers / 2026 / MemBuilder: Reinforcing LLMs for Long-Term Memory Construction via Attributed Dense Rewards

MemBuilder: Reinforcing LLMs for Long-Term Memory Construction via Attributed Dense Rewards

Paper · 2026 · Zhiyu Shen, Ziming Wu, Fuming Lai et al.

Summary

Long multi-session conversations are hard for LLMs because plain retrieval treats past text as isolated chunks and misses how facts change over time. Existing memory frameworks such as Mem0, MIRIX and MemGPT use prompts on costly closed-source models and get no feedback on whether their stored memories help. RL-trained alternatives such as Memory-R1 and Mem-α rely on a single coarse reward. MemBuilder trains one Qwen3-4B model to maintain four kinds of memory: Core, Episodic, Semantic and Procedural. Training starts with supervised fine-tuning on expert trajectories produced by Claude 4.5 Sonnet. It then applies ADRPO (Attributed Dense Rewards Policy Optimization). ADRPO rewards the model after every session using synthetic questions about that session. It also weights each memory type's gradient by how often that type is retrieved when answering. Although trained only on LongMemEval, the model reaches 84.23% on LoCoMo and beats Claude 4.5 Sonnet under identical settings. This shows that a small open model can take over the expensive memory-building step.

Key points

  • Memory operations: Episodic memory supports UPDATE and MERGE, which keep earlier entries instead of deleting them. This preserves a timeline of how information evolved.
  • Gains over Memory-R1 on LoCoMo: MultiHop rises from 53.01% to 77.88%, Temporal from 51.55% to 71.71%, and Adversarial reaches 90.58%.
  • Dense rewards matter: When only 1/8 of sessions receive rewards, accuracy drops to 80.92%. That is below the SFT-only baseline of 81.74%.
  • Attribution checks: The best gradient-weighting setting is α = 4. Human checks found retrieval frequency matched the true information source in 95% of cases.
  • Behaviour and portability:
    • After RL, Core memory shifts from mostly APPEND (98.1% → 12.9%) to mostly REPLACE (87.1%).
    • The memory works with other answer models. Using the trained Qwen3-4B as the answerer scores 81.12% on LoCoMo, versus 74.61% for the base model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China134 works · 19,743 citations🎓Sun Yat-sen UniversityGuangzhou · China53 works · 9,185 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Zhiyu Shen, Ziming Wu, Fuming Lai, Shaobing Lian and Yanghui Rao. 2026. MemBuilder: Reinforcing LLMs for Long-Term Memory Construction via Attributed Dense Rewards. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1284

@inproceedings{shen2026membuilder,
  title = {{MemBuilder: Reinforcing LLMs for Long-Term Memory Construction via Attributed Dense Rewards}},
  author = {Zhiyu Shen, Ziming Wu, Fuming Lai, Shaobing Lian and Yanghui Rao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1284},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1284}
}

References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. Longalign: A recipe for long context alignment of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1376–1395.
  2. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023. Longlora: Efficient fine-tuning of long-context large language models. CoRR, abs/2309.12307.
  3. Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building production-ready AI agents with scalable long-term memory. CoRR, abs/2504.19413.
  4. Yiming Du, Hongru Wang, Zhengyi Zhao, Bin Liang, Baojun Wang, Wanjun Zhong, Zezhong Wang, and Kam-Fai Wong. 2024. PerLTQA: A personal long-term memory dataset for memory classification, retrieval, and fusion in question answering. In Proceedings of the 10th SIGHAN Workshop on Chinese Language Proces… find
  5. Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. 2018. Counterfactual multi-agent policy gradients. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), pages 2974– 2982. find
  6. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey. Preprint, arXiv:2312.10997.
  7. John E. Laird. 2012. The Soar Cognitive Architecture. find
  8. Yueheng Li, Guangming Xie, and Zongqing Lu. 2022. Difference advantage estimation for multi-agent policy gradients. In Proceedings of the 39th International Conference on Machine Learning (ICML 2022), volume 162 of Proceedings of Machine Learning Research, pages 13066–13085. find
  9. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  10. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok… find
  11. Mastra. 2025. Yes, you can use rag for agent memory. https://mastra.ai/blog/ use-rag-for-agent-memory. Accessed: 2025. ↗
  12. Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. 2023. Memgpt: Towards llms as operating systems. CoRR, abs/2310.08560. find
  13. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024. find
  14. Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2018. Learning to reweight examples for robust deep learning. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018), volume 80 of Proceedings of Machine Learning Research, pages 4334–4343. find
  15. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300.
  16. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 20…
  17. Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. 2024. Cognitive architectures for language agents. Transactions on Machine Learning Research. find
  18. Zhen Tan, Tianshu Shen, I-Hung Hsu, Anbang Xu, Longju Bai, Yash Jain, Chen-Yu Lee, Hamid Palangi, Tianlong Chen, Long T. Le, and Rujun Han. 2025. In prospect and retrospect: Reflective memory management for long-term personalized dialogue agents. In Proceedings of the 63rd Annual Meeting of the Asso…
  19. Endel Tulving. 1972. Episodic and semantic memory. In Endel Tulving and Wayne Donaldson, editors, Organization of Memory, pages 381–403. find
  20. Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Milos. 2023. Focused transformer: Contrastive training for context scaling. In Advances in Neural Information Processing Systems 36: NeurIPS 2023. find
  21. Bing Wang, Xinnian Liang, Jian Yang, Hui Huang, Zhenhe Wu, ShuangZhi Wu, Zejun Ma, and Zhoujun Li. 2026. Scm: Enhancing large language model with self-controlled memory framework. In Database Systems for Advanced Applications, pages 188–203, Singapore. find
  22. Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. 2023. Augmenting language models with long-term memory. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA,… find
  23. Yu Wang and Xi Chen. 2025. MIRIX: multi-agent memory system for llm-based agents. CoRR, abs/2507.07957. — [arXiv:2507.07957](https://arxiv.org/abs/2507.07957) find
  24. Yu Wang, Ryuichi Takanobu, Zhiqi Liang, Yuzhen Mao, Yuanzhe Hu, Julian J. McAuley, and Xiaojian Wu. 2025. Mem-α: Learning memory construction via reinforcement learning. CoRR, abs/2509.25911. — [arXiv:2509.25911](https://arxiv.org/abs/2509.25911) find
  25. Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. 2025. Longmemeval: Benchmarking chat assistants on long-term interactive memory. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025.
  26. Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Hinrich Schütze, Volker Tresp, and Yunpu Ma. 2025. Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning. CoRR, abs/2508.19828.
  27. Chong Zhang, Yue Deng, Xiang Lin, Bin Wang, Dianwen Ng, Hai Ye, Xingxuan Li, Yao Xiao, Zhanfeng Mo, Qi Zhang, and Lidong Bing. 2025a. 100 days after deepseek-r1: A survey on replication studies and more directions for reasoning language models. Preprint, arXiv:2505.00551.
  28. Gaoke Zhang, Bo Wang, Yunlong Ma, Dongming Zhao, and Zifei Yu. 2025b. Multiple memory systems for enhancing the long-term memory of agent. CoRR, abs/2508.15294. — [arXiv:2508.15294](https://arxiv.org/abs/2508.15294) find
  29. Guibin Zhang, Muxin Fu, and Shuicheng Yan. 2025c. Memgen: Weaving generative latent memory for self-evolving agents. CoRR, abs/2509.24704. — [arXiv:2509.24704](https://arxiv.org/abs/2509.24704) find
  30. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400–410, Bangkok, Thail…
  31. Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memorybank: Enhancing large language models with long-term memory. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fo… find
  32. Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. 2025. MEM1: learning to synergize memory and reasoning for efficient long-horizon agents. CoRR, abs/2506.15841. — [arXiv:2506.15841](https://arxiv.org/abs/2506.15841) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2026-membuilder-reinforcing-llms-long-term/