Elamur Research⌕
☰
home / papers / 2026 / From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents

From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents

Paper · 2026 · Md Nayem Uddin, Kumar Shubham, Eduardo Blanco et al.

Summary

Most benchmarks for long-term conversational memory reduce the problem to retrieving a single fact from an earlier session. They seldom test whether an agent can combine information spread over many sessions or keep up with repeated revisions and deletions of user information. The authors address this with Memora, a simulated benchmark. Ten professional personas generate user–assistant conversations lasting a week, a month or a quarter. Three task types are built on this history: remembering, reasoning and recommending, and every answer must respect which information is currently valid. Data quality is checked with automated memory-grounding tests and human evaluation. The authors also propose FAMA (Forgetting-Aware Memory Accuracy), which rewards correct use of valid memory and deducts credit when a response depends on outdated or deleted information. They test four LLMs and six memory agents. All systems often reuse invalidated memories, and dedicated memory agents improve only slightly over plain LLMs. Reasoning is the weakest task. The work matters because it shows that personalized assistants struggle with keeping memory current, a problem standard inclusion-based metrics hide.

Key points

  • Higher demands than earlier benchmarks. Prior benchmarks (PerLTQA, MemDaily, LOCOMO, LongMemEval, PersonaMem) need about 1–3 sessions of context per query on average. Memora needs 5.3 (weekly), 17.3 (monthly) and 28.4 (quarterly), with a maximum of 309. It averages 2.7, 8.8 and 14.8 updates or deletions per query, with a maximum of 94.
  • Earlier benchmarks rely on shallow recall. In LoCoMo, 94% of questions draw on at most two prior sessions. The same holds for 85% of LongMemEval questions.
  • Agents evaluated. The evaluation covers six dedicated memory systems: A-Mem, LangMem, Mem-0, MemoBase, MemoryOS and Nemori.
  • FAMA changes rankings. In the monthly setting, Nemori scores below MemoryOS and A-Mem on plain memory-presence accuracy. It loses far less under the forgetting penalty (15.4 vs. 28.4 and 29.5 points), so it ranks above both.
  • Error analysis (75 sampled errors, 25 per task).
    • Recommending: 64% of errors come from failing to discard outdated preferences.
    • Remembering: 72% come from partial retrieval.
    • Reasoning: every error involves incomplete retrieval, which blocks correct consolidation.
  • Availability. Code and data are released at github.com/geniesinc/Memora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Arizona State UniversityTempe · United States50 works · 2,032 citations🎓University of ArizonaTucson · United States38 works · 10,603 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Md Nayem Uddin, Kumar Shubham, Eduardo Blanco, Chitta Baral and Gengyu Wang. 2026. From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1337

@inproceedings{uddin2026recall,
  title = {{From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents}},
  author = {Md Nayem Uddin, Kumar Shubham, Eduardo Blanco, Chitta Baral and Gengyu Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1337},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1337}
}

References

44 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Associat… find
  2. Pedro Bekinschtein, Noelia V Weisstaub, Francisco Gallo, Maria Renner, and Michael C Anderson. 2018. A retrieval-specific mechanism of adaptive forgetting in the mammalian brain. Nature Communications, 9(1):4660. find
  3. Sarah Brown-Schmidt, Christopher Brett Jaeger, Kaitlin Lord, and Aaron S Benjamin. 2025. Remembering conversation in group settings. Memory & Cognition, 53(4):1037–1054. find
  4. Eason Chen, Jia-En Lee, Jionghao Lin, and Kenneth Koedinger. 2024. Gptutor: Great personalized tutor with large language models for personalized learning content generation. find
  5. Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413.
  6. Kaustubh Deshpande, Ved Sirdeshmukh, Johannes Baptist Mols, Lifeng Jin, Ed-Yeremai Hernandez-Cardona, Dean Lee, Jeremy Kritz, Willow E. Primack, Summer Yue, and Chen Xing. 2025. Multi-Challenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier LLMs. In Findings of the… find
  7. Yiming Du, Hongru Wang, Zhengyi Zhao, Bin Liang, Baojun Wang, Wanjun Zhong, Zezhong Wang, and Kam-Fai Wong. 2024. PerLTQA: A personal long-term memory dataset for memory classification, retrieval, and fusion in question answering. In Proceedings of the 10th SIGHAN Workshop on Chinese Language Proces… find
  8. Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, and Hao Peng. 2025. Context length alone hurts LLM performance despite perfect retrieval. In Findings of the Association for Computational Linguistics: EMNLP 20… find
  9. K Anders Ericsson and Walter Kintsch. 1995. Long-term working memory. Psychological review, 102(2):211. find
  10. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians,…
  11. Robin M Hogarth and Hillel J Einhorn. 1992. Order effects in belief updating: The belief-adjustment model. Cognitive Psychology, 24(1):1–55. find
  12. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models?
  13. Yuanzhe Hu, Yu Wang, and Julian McAuley. 2025. Evaluating memory in llm agents via incremental multi-turn interactions. find
  14. Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065, Toronto, Canada. Association for Computational Linguistics.
  15. Bowen Jiang, Zhuoqun Hao, Young-Min Cho, Bryan Li, Yuan Yuan, Sihao Chen, Lyle Ungar, Camillo J Taylor, and Dan Roth. 2025. Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale. arXiv preprint arXiv:2504.14225. — [arXiv:2504.14225](https://arxiv.org/abs/2504.14225) find
  16. Natalie A Jones, Helen Ross, Timothy Lynam, Pascal Perez, and Anne Leitch. 2011. Mental models: an interdisciplinary synthesis of theory and methods. Ecology and society, 16(1). find
  17. Jiho Kim, Woosog Chay, Hyeonji Hwang, Daeun Kyung, Hyunseung Chung, Eunbyeol Cho, Yeonsu Kwon, Yohan Jo, and Edward Choi. 2025. Dialsim: A dialogue simulator for evaluating long-term multi-party dialogue understanding of conversational agents. find
  18. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  19. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–9474.
  20. Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang, Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, et al. 2024. Controllable text generation for large language models: A survey. arXiv preprint arXiv:2408.12599.
  21. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  22. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851– 1387… find
  23. Mark E Mazurek, Jamie D Roitman, Jochen Ditterich, and Michael N Shadlen. 2003. A role for neural integrators in perceptual decision making. Cerebral cortex, 13(11):1257–1269. find
  24. Martijn Meeter and Jaap MJ Murre. 2004. Consolidation of long-term memory: evidence and alternatives. Psychological Bulletin, 130(6):843. find
  25. Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, et al. 2025. A survey of context engineering for large language models. arXiv preprint arXiv:2507.13334. — [arXiv:2507.13334](https://arxiv.org/abs/2507.13334) find
  26. Jiayan Nan, Wenquan Ma, Wenlong Wu, and Yize Chen. 2025. Nemori: Self-organizing agent memory inspired by cognitive science. arXiv preprint arXiv:2508.03341. — [arXiv:2508.03341](https://arxiv.org/abs/2508.03341) find
  27. Charles Packer, Vivian Fang, Shishir_G Patil, Kevin Lin, Sarah Wooders, and Joseph_E Gonzalez. 2023. Memgpt: Towards llms as operating systems. find
  28. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. find
  29. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  30. Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxing Xu, et al. 2024. Benchmarking complex instruction-following with multiple constraints composition. Advances in Neural Information Processing Systems, 37:137610–137645. find
  31. Rachel Wood, Paul Baxter, and Tony Belpaeme. 2012. A review of long-term memory in natural and synthetic systems. Adaptive Behavior, 20(2):81–103. find
  32. Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. 2024. Longmemeval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410.10813.
  33. Yaxiong Wu, Sheng Liang, Chen Zhang, Yichao Wang, Yongyue Zhang, Huifeng Guo, Ruiming Tang, and Yong Liu. 2025. From human memory to ai memory: A survey on memory mechanisms in the era of llms. find
  34. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244.
  35. Jing Xu, Arthur Szlam, and Jason Weston. 2022a. Beyond goldfish memory: Long-term open-domain conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5180–5197, Dublin, Ireland. Association for Computational Linguistics. find
  36. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025. A-mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110.
  37. Xinchao Xu, Zhibin Gou, Wenquan Wu, Zheng-Yu Niu, Hua Wu, Haifeng Wang, and Shihang Wang. 2022b. Long time no see! open-domain conversation with long-term persona memory. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2639–2650, Dublin, Ireland. Association for Computa… find
  38. Zhifang Ye, Liang Shi, Anqi Li, Chuansheng Chen, and Gui Xue. 2020. Retrieval practice facilitates memory updating by enhancing and differentiating medial prefrontal cortex representations. Elife, 9:e57023. find
  39. Ruifeng Yuan, Shichao Sun, Yongqi Li, Zili Wang, Ziqiang Cao, and Wenjie Li. 2025. Personalized large language model assistant with evolving conditional memory. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3764–3777, Abu Dhabi, UAE. Association for Computat… find
  40. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024a. ∞Bench: Extending long context evaluation beyond 100K tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu… find
  41. Yutong Zhang, Dora Zhao, Jeffrey T. Hancock, Robert Kraut, and Diyi Yang. 2025. The rise of ai companions: How human-chatbot relationships influence well-being. find
  42. Zeyu Zhang, Quanyu Dai, Luyu Chen, Zeren Jiang, Rui Li, Jieming Zhu, Xu Chen, Yi Xie, Zhenhua Dong, and Ji-Rong Wen. 2024b. Memsim: A bayesian simulator for evaluating memory of llm-based personal assistants. arXiv preprint arXiv:2409.20163. — [arXiv:2409.20163](https://arxiv.org/abs/2409.20163) find
  43. Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2023. Memorybank: Enhancing large language models with long-term memory. find
  44. Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/uddin-2026-recall-forgetting-benchmarking-long-term/