Elamur Research⌕
☰
home / papers / 2026 / Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents

Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents

Paper · 2026 · Yiting Shen, Kun Li, Wei Zhou et al.

Summary

Most memory benchmarks for LLM agents check whether a system can recall a stored fact when asked directly. Mem2ActBench targets a harder skill. A user gives a brief, underspecified request, and the agent must work out which earlier preferences or task states apply, then use them to choose the right tool and fill in its arguments.

The authors build long, interrupted interaction histories with an automated pipeline. It combines tool-use dialogues from ToolACE and BFCL with off-topic chat from OASST1. Facts are then extracted, grouped and ordered into a conflict-free "fact evolution chain." Tasks are created backwards from ground-truth tool calls, and a leakage check makes sure the final query alone cannot reveal the answer.

Tests on seven memory frameworks show that tool selection is generally reliable. Filling in parameters from memory remains weak, especially when the evidence sits in the middle of the history. The benchmark therefore exposes a gap that matters for long-lived assistants.

Key points

  • The resource contains 2,029 synthesized sessions, averaging about 12 user–assistant–tool turns each, and 400 memory-dependent tool-use tasks. Human checking found 91.3% of tasks truly require long-term memory.
  • The pipeline turned 8,000 ToolACE samples and BFCL_v3 subsets (live, parallel, multi-turn) into dialogues. It added top-ranked OASST1 threads as noise and grouped extracted facts with BERTopic using an HDBSCAN backend.
  • Errors fall into five categories. Retrieval Miss is the most common even for strong systems. As frameworks improve, more failures shift to evidence that is retrieved but then not used.
  • Three mitigations were tested with Qwen2.5-72B-Instruct on 200 tasks, against an LTMemory baseline with Parameter F1 of 29.24:
    • Query Expansion raised Parameter F1 to 32.42.
    • Self-Refine raised tool accuracy to 94% but barely helped arguments.
    • Interactive Clarification with an oracle user reached an F1 of 48.68, which the authors treat as a diagnostic upper bound.
  • Code and data are available at github.com/Cantaloupe-M/Mem2ActBench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🔬Institute of Information EngineeringBeijing · China30 works · 1,411 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Yiting Shen, Kun Li, Wei Zhou and Songlin Hu. 2026. Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.370

@inproceedings{shen2026mem2actbench,
  title = {{Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents}},
  author = {Yiting Shen, Kun Li, Wei Zhou and Songlin Hu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.370},
  url = {https://doi.org/10.18653/v1/2026.acl-long.370}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics ACL 2024, pages 2318–2335.
  2. Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. 2025. Mem0: Building production-ready ai agents with scalable long-term memory. Preprint, arXiv:2504.19413.
  3. Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794.
  4. Yuanzhe Hu, Yu Wang, and Julian McAuley. 2025. Evaluating memory in llm agents via incremental multi-turn interactions. arXiv preprint arXiv:2507.05257. — [arXiv:2507.05257](https://arxiv.org/abs/2507.05257) find
  5. Jiho Kim, Woosog Chay, Hyeonji Hwang, Daeun Kyung, Hyunseung Chung, Eunbyeol Cho, Yeonsu Kwon, Yohan Jo, and Edward Choi. 2025. Dialsim: A dialogue simulator for evaluating long-term multi-party dialogue understanding of conversational agents. Preprint, arXiv:2406.13144. — [arXiv:2406.13144](https://arxiv.org/abs/2406.13144) find
  6. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023.… — OpenAssistant Conversations — Democratizing Large Language Model Alignment
  7. LangChain. 2025. Langmem (langchain-ai/langmem). https://github.com/langchain-ai/langmem. Version 0.0.30; accessed 2026-01-03. ↗
  8. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  9. Weiwen Liu, Xu Huang, Xingshan Zeng, xinlong hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong WANG, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Wang Xinzhi, Yong Liu, Yasheng Wang, and 8 others. 2025. Toolace: Winning the points of llm function calling.… find
  10. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of llm agents. arXiv preprint arXiv:2402.17753. find
  11. Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Schütze. 2024. Ret-llm: Towards a general read-write memory for large language models. Preprint, arXiv:2305.14322. — [arXiv:2305.14322](https://arxiv.org/abs/2305.14322) find
  12. Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Memgpt: Towards llms as operating systems. Preprint, arXiv:2310.08560. find
  13. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22. find
  14. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  15. Alireza Rezazadeh, Zichao Li, Wei Wei, and Yujia Bao. 2024. From isolated conversations to hierarchical schemas: Dynamic tree memory representation for llms. arXiv preprint arXiv:2410.14052. — [arXiv:2410.14052](https://arxiv.org/abs/2410.14052) find
  16. Bing Wang, Xinnian Liang, Jian Yang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. 2023. Scm: Enhancing large language model with self-controlled memory framework. arXiv e-prints, pages arXiv–2304. find
  17. Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. 2024. Longmemeval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410.10813.
  18. Jing Xu, Arthur Szlam, and Jason Weston. 2022. Beyond goldfish memory: Long-term open-domain conversation. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 5180–5197. find
  19. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, A-Juntao Tan, and Yongfeng Zhang. 2025. mem: Agentic memory for llm agents. Preprint, arXiv:2502.12110.
  20. Sanae Yamashita, Koji Inoue, Ao Guo, Shota Mochizuki, Tatsuya Kawahara, and Ryuichiro Higashinaka. 2023. Realpersonachat: A realistic persona chat corpus with interlocutors own personalities. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, pages 852–861. find
  21. Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shen-2026-mem2actbench-benchmark-evaluating-long-term/