Elamur Research⌕
☰
home / papers / 2026 / H-MEM: Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents

H-MEM: Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents

Paper · 2026 · Haoran Sun, Shaoning Zeng and Bob Zhang

Summary

LLM agents in long conversations must draw on past interactions, but common memory designs fall short. Flat vector stores such as MemoryBank compare each query against every stored entry, so cost grows quickly as memory accumulates, and graph-style memories are hard to keep consistent. H-MEM addresses this by arranging memory as a four-level hierarchy ordered by semantic abstraction: Domain, Category, Memory Trace and Episode, comparable to the section levels of a document. After each turn, a prompted extraction model fills these levels. The bottom level holds the full episode with a timestamp and an inferred user profile. Each memory vector carries position indices pointing to its children one level down, so retrieval descends level by level and skips unrelated branches rather than scanning everything. On five question-answering task types from LoCoMo, H-MEM beats five baselines across several LLMs and needs far less retrieval computation and time than MemoryBank. This makes long-term agent memory both more accurate and more scalable.

Key points

  • The memory hierarchy has four layers: Domain, Category, Memory Trace and Episode. The top three act as an index. The Episode layer keeps both the vectors and the raw text, plus a user profile. The number of levels can be changed through an adjustment interface.
  • Retrieval uses index-based routing through each memory's sub-memory position encodings, which avoids exhaustive similarity search over all entries.
  • H-MEM was compared with five baseline methods on five LoCoMo QA task settings and outperformed them consistently.
  • In efficiency tests with Qwen-1.5B, retrieval cost for the MemoryBank baseline rose almost exponentially as memory grew, while H-MEM's cost leveled off. At maximum memory load, the baseline took over 400 ms, about five times slower than H-MEM.
  • Ablations on Qwen-1.5B removed the retrieval component alone (w/o R.) and then both storage and retrieval (w/o H&R.). Performance dropped at each step, which indicates that the hierarchical storage and the index retrieval depend on each other. The authors name multimodal memory as a limitation and a direction for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MacauMacao · China33 works · 2,223 citations🎓University of Electronic Science and Technology of ChinaChengdu · China31 works · 2,012 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Haoran Sun, Shaoning Zeng and Bob Zhang. 2026. H-MEM: Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.15

@inproceedings{sun2026h,
  title = {{H-MEM: Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents}},
  author = {Haoran Sun, Shaoning Zeng and Bob Zhang},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.15},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.15}
}

References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  2. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The faiss library.
  3. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. find
  4. Kostas Hatalis, Despina Christou, Joshua Myers, Steven Jones, Keith Lambert, Adam Amos-Binks, Zohreh Dannenhauer, and Dustin Dannenhauer. 2023. Memory matters: The need to improve long-term memory in llm-agents. In Proceedings of the AAAI Symposium Series, volume 2, pages 277–280. find
  5. Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024a. Understanding the planning of llm agents: A survey. arXiv preprint arXiv:2402.02716. find
  6. Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024b. Understanding the planning of llm agents: A survey. arXiv preprint arXiv:2402.02716. find
  7. Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John Canny, and Ian Fischer. 2024. A human-inspired reading agent with gist memory of very long contexts. Preprint, arXiv:2402.09727.
  8. Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, and 1 others. 2024. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459. find
  9. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating very long-term conversational memory of llm agents. arXiv preprint arXiv:2402.17753. find
  10. Charles Packer, Vivian Fang, Shishir_G Patil, Kevin Lin, Sarah Wooders, and Joseph_E Gonzalez. 2023. Memgpt: Towards llms as operating systems. find
  11. Rana Salama, Jason Cai, Michelle Yuan, Anna Currey, Monica Sunkara, Yi Zhang, and Yassine Benajiba. 2025. Meminsight: Autonomous memory augmentation for llm agents. arXiv preprint arXiv:2503.21760.
  12. Zhuocheng Shen. 2024. Llm with tools: A survey. arXiv preprint arXiv:2409.18807. — [arXiv:2409.18807](https://arxiv.org/abs/2409.18807) find
  13. Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2024a. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404.13501. find
  14. Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2024b. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404.13501. find
  15. Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731. find
  16. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint…
  17. Bing Wang, Xinnian Liang, Jian Yang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. 2023. Enhancing large language model with self-controlled memory framework. arXiv preprint arXiv:2304.13343. — [arXiv:2304.13343](https://arxiv.org/abs/2304.13343) find
  18. Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, and Derek Fai Wong. 2025. A survey on llm-generated text detection: Necessity, methods, and future directions. Computational Linguistics, pages 1–66.
  19. Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. 2025. A-mem: Agentic memory for llm agents. arXiv preprint arXiv:2502.12110.
  20. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 40 others. 2024a. Qwen2 technical report. arXiv preprint arX… find
  21. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024b. Qwen2.5 technical report. arXiv preprint arX… find
  22. Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, page 100211.
  23. Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. 2024. A survey on recent advances in llm-based multi-turn dialogue systems. arXiv preprint arXiv:2402.18013.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-h-mem-hierarchical-memory-high/