Elamur Research⌕
☰
home / papers / 2026 / StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall

StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall

Paper · 2026 · Yerong Wu, Tianxing Wu, Minghao Zhu et al.

Summary

Most benchmarks for memory-augmented dialogue measure whether a model can retrieve and restate stored facts. This paper argues that believable virtual characters need more than that. They must judge which memories to bring into a reply. The authors sort each instance's memories into three roles: must (needed for a correct answer), nice (optional material that adds personalization or empathy) and irr (unrelated material that should stay out). These roles are hidden from the model, so it has to infer relevance from the persona, dialogue history and query.

The authors derived StratMem-Bench (657 instances) from LoCoMo. They also defined four metrics to score memory compliance, integration quality, proactive enrichment and irrelevant intrusions. When state-of-the-art LLMs played the characters, they reliably used required memories and ignored irrelevant ones. Their decisions became inconsistent once supportive memories were in the pool. Models that enriched replies more also leaked more irrelevant content. The benchmark reframes memory as a resource used strategically in conversation, not a store for factual lookup.

Key points

  • Dataset construction: StratMem-Bench has 657 instances built from LoCoMo multi-session dialogues and summaries.
    • A sliding window of 1–3 sessions forms each memory pool.
    • The dialogue history comes from the session that follows, which keeps memory acquisition and response generation separate in time.
  • Metrics: four new measures are introduced:
    • Strict Memory Compliance (SMC)
    • Memory Integration Quality (MIQ)
    • Proactive Enrichment Score (PES)
    • Conditional Irrelevance Rate (CIR)
  • Integration quality (MIQ, 1–5 scale):
    • Replies using must memories score about 4.2–4.6.
    • Replies using nice memories score lower, about 3.9–4.4; the authors call this gap an "enrichment tax."
    • Replies using irr memories collapse to about 2.6–3.1.
  • Proactivity–risk trade-off:
    • GPT-5-chat has the lowest CIR (≈7.9%) but enriches replies conservatively.
    • Gemini 3 Pro reaches the highest PES (≈73.3%), but its CIR rises to about 47% in nice-only scenarios.
  • Models and limitations: the evaluated models include GPT-5.2, Claude Sonnet 4.5, DeepSeek-chat/reasoner, Llama 4 Maverick, Qwen3-Max and Qwen3-235B, and none clearly dominates. The benchmark covers only single-turn, text-only memory use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksSocial Psychology · 330 works

Cite

Cite as Yerong Wu, Tianxing Wu, Minghao Zhu, Hangyu Sha and Haofen Wang. 2026. StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1491

@inproceedings{wu2026stratmem,
  title = {{StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall}},
  author = {Yerong Wu, Tianxing Wu, Minghao Zhu, Hangyu Sha and Haofen Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1491},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1491}
}

References

35 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicole Alea and Susan Bluck. 2003. Why are you telling me that? a conceptual model of the social function of autobiographical memory. Memory, 11(2):165–178. find
  2. John R. Anderson. 1983. The Architecture of Cognition. Harvard University Press. find
  3. Michael C. Anderson and Collin Green. 2001. Suppressing unwanted memories by executive control. Nature, 410:366–369. find
  4. Anthropic. 2025. Claude sonnet 4.5 system card. https://assets.anthropic.com/m/12f214efcc2f457a/original/ Ac-Claude-Sonnet-4-5-System-Card.pdf. cessed: 2025-12-15. ↗
  5. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  6. Allan M. Collins and Elizabeth F. Loftus. 1975. A spreading-activation theory of semantic processing. Psychological Review, 82(6):407–428. find
  7. DeepSeek-AI. 2025. Deepseek-v3.2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  8. Bowen Jiang, Zhuoqun Hao, Young-Min Cho, Bryan Li, Yuan Yuan, Sihao Chen, Lyle Ungar, Camillo J Taylor, and Dan Roth. 2025. Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale. arXiv preprint arXiv:2504.14225. — [arXiv:2504.14225](https://arxiv.org/abs/2504.14225) find
  9. Yukyung Lee, Joonghoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pilsung Kang, and Najoung Kim. 2025. Checkeval: A reliable llm-as-a-judge framework for evaluating text generation using checklists. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1578…
  10. Xiaopeng Li, Pengyue Jia, Derong Xu, Yi Wen, Yingyi Zhang, Wenlin Zhang, Wanyu Wang, Yichao Wang, Zhaocheng Du, Xiangyang Li, and 1 others. 2025. A survey of personalization: From rag to agent. arXiv preprint arXiv:2504.10147. — [arXiv:2504.10147](https://arxiv.org/abs/2504.10147) find
  11. Alexander R Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. find
  12. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 14017–14046, Bangkok, Thailand. Ass… find
  13. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  14. Thomas O. Nelson and Louis Narens. 1990. Metamemory: A theoretical framework and new findings. Psychology of Learning and Motivation, 26:125–173. find
  15. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2(1).
  16. Man Tik Ng, Hui Tung Tse, Jen-tse Huang, Jingjing Li, Wenxuan Wang, and Michael R Lyu. 2024. How well can llms echo us? evaluating ai chatbots’ role-play ability with echo. arXiv preprint arXiv:2404.13957. — [arXiv:2404.13957](https://arxiv.org/abs/2404.13957) find
  17. Google. 2025. Gemini 3 developer guide (gemini api): Gemini 3 pro preview. https://ai.google. Model ID: dev/gemini-api/docs/gemini-3. gemini-3-pro-preview. Accessed: 2025-12-15. ↗
  18. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. find
  19. Junqing He, Liang Zhu, Rui Wang, Xi Wang, Gholamreza Haffari, and Jiaxing Zhang. 2025. MADial-bench: Towards real-world evaluation of memory-augmented dialogue generation. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang… find
  20. Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty. 2025. Multilingual needle in a haystack: Investigating long-context behavior of multilingual large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lingui… find
  21. OpenAI. 2025. Gpt-5.1 technical report. https://openai.com/research. Accessed: 2025-12-5. ↗
  22. Monisha Pasupathi. 2001. The social construction of the personal past and its implications for adult development. Psychological Bulletin, 127(5):651–672. find
  23. David Rau, Hervé Déjean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, Stéphane Clinchant, and Vassilina Nikoulina. 2024. Bergen: A benchmarking library for retrieval-augmented generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7640–7663. find
  24. Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik Narasimhan, and 1 others. 2024. Personagym: Evaluating persona agents and llms. arXiv preprint arXiv:2407.18416. find
  25. Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-llm: A trainable agent for role-playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
  26. Teng Shi, Jun Xu, Xiao Zhang, Xiaoxue Zang, Kai Zheng, Yang Song, and Han Li. 2025. Retrieval augmented generation with collaborative filtering for personalized text generation. arXiv:2504.05731. arXiv preprint Quan Tu, Shilong Fan, Zihang Tian, and Rui Yan. 2024. Charactereval: A chinese benchmark…
  27. Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, and 1 others. 2024a. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Findings of the Association for Computation…
  28. Zheng Wang, Zhongyang Li, Zeren Jiang, Dandan Tu, and Wei Shi. 2024b. Crafting personalized agents through retrieval-augmented generation on editable memory graphs. arXiv preprint arXiv:2409.19401. find
  29. Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2024. Self-preference bias in llm-as-a-judge. arXiv preprint arXiv:2410.21819.
  30. Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. 2024. Longmemeval: Benchmarking chat assistants on long-term interactive memory. arXiv preprint arXiv:2410.10813.
  31. An Yang, Binyuan Hui, Jian Yang, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  32. Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze D Gui, Ziran W Jiang, Ziyu Jiang, and 1 others. 2024. Crag-comprehensive rag benchmark. Advances in Neural Information Processing Systems, 37:10470–10490. find
  33. S. Zhao and 1 others. 2024a. Do llms recognize your preferences? evaluating personalized preference following in llms. OpenReview. find
  34. Suifeng Zhao, Tong Zhou, Zhuoran Jin, Hongbang Yuan, Yubo Chen, Kang Liu, and Sujian Li. 2024b. AWeCita: Generating Answer with Appropriate and Well-grained Citations Using LLMs. Data Intelligence, 6(4):1134–1157. find
  35. Zheng Zhao, Clara Vania, Subhradeep Kayal, Naila Khan, Shay B. Cohen, and Emine Yilmaz. 2025. Per-sonalens: A benchmark for personalization evaluation in conversational ai assistants. In Findings of ACL. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-stratmem-bench-evaluating-strategic-memory/