Elamur Research⌕
☰
home / papers / 2026 / Agentic Episodic Control

Agentic Episodic Control

Paper · 2026 · Xidong Yang, Wenhao Li, Junjie Sheng et al.

Summary

Episodic reinforcement learning agents store past experiences in an external memory to learn faster. The authors argue that such agents have two weaknesses. First, small trainable encoders produce narrow state embeddings, which they call a representation bottleneck. Second, the agent queries memory with KNN at every step, a retrieval dilemma; under sparse rewards, these lookups mostly return low-value neighbors.

Agentic Episodic Control (AEC) brings LLMs into the episodic control loop through three components:

  • a semantic augmenter that rewrites raw text observations into language-based abstractions, which serve as memory keys;
  • a short-term working memory that summarizes the current context to support exploration;
  • a critical-state recognizer that decides whether recalling memory is worthwhile at a given step.

On five BabyAI-Text tasks, AEC learns 2–6 times more data-efficiently than baselines. It is the only tested method that reliably solves UnlockLocal, and it generalizes across tasks and environments under distribution shift. The work suggests that LLM priors can make RL agents more sample-efficient and adaptable, though LLM calls add inference cost.

Key points

  • Benchmarks: GoToLocal, PickupDist, PickupLocal, UnlockLocal and FindObj from BabyAI-Text. AEC exceeds 90% success on UnlockLocal.
  • Representation test (300 states, balanced over three optimal actions):
    • SentenceTransformer embeddings of LLM-augmented states give over 95% KNN action-consistency at k=1 and stay above 92% at k=10.
    • A DRRN encoder trained for 70K frames barely beats an untrained one.
  • Selective retrieval: querying memory at every step lowers success, most sharply on UnlockLocal.
  • Ablations: removing either the augmenter or the working memory hurts performance on all tasks. Working memory matters most on the very sparse-reward FindObj.
  • Backbones and limitations:
    • The main backbone is Qwen2.5-32B-Instruct.
    • Qwen2.5-7B-Instruct still works, though somewhat worse.
    • The main limitation is higher inference latency from LLM queries.
    • Code is released on GitHub (Xidong-Yang/Agentic_Episodic_Control).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🎓East China Normal UniversityShanghai · China69 works · 2,490 citations🎓Tongji UniversityShanghai · China33 works · 2,993 citations

Also on the byline: Shanghai University of International Business and Economics.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Xidong Yang, Wenhao Li, Junjie Sheng, Yun Hua, Haosheng Chen, Chuyun Shen and Xiangfeng Wang. 2026. Agentic Episodic Control. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.654

@inproceedings{yang2026agentic,
  title = {{Agentic Episodic Control}},
  author = {Xidong Yang, Wenhao Li, Junjie Sheng, Yun Hua, Haosheng Chen, Chuyun Shen and Xiangfeng Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.654},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.654}
}

References

50 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vinamra Benara, Chandan Singh, John X. Morris, Richard J. Antonello, Ion Stoica, Alexander Huth, and Jianfeng Gao. 2024. Crafting interpretable embeddings for language neuroscience by asking llms questions. In Advances in Neural Information Processing Systems. find
  2. Howard Eichenbaum. 2004. Hippocampus: Cognitive processes and neural representations that underlie declarative memory. Neuron, 44(1):109–120. find
  3. Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J R. Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, and 1 others. 2022. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 61… find
  4. Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomènech Badia, Gavin Buttimore, Charles Deck, Joel Z Leibo, and Charles Blundell. 2019. Generalization of reinforcement learners with working and episodic memory. In Advances in Neural Information Processing Systems. find
  5. Steven Hansen, Alexander Pritzel, Pablo Sprechmann, André Barreto, and Charles Blundell. 2018. Fast deep reinforcement learning using online adjustments from the past. In Advances in Neural Information Processing Systems. find
  6. Siddhant Bhambri, Amrita Bhattacharjee, Subbarao Kambhampati, and 1 others. 2024. Efficient reinforcement learning via large language model-based search. In NeurIPS 2024 Workshop on Open-World Agents. find
  7. Ji He, Mari Ostendorf, Xiaodong He, Jianshu Chen, Jianfeng Gao, Lihong Li, and Li Deng. 2016. Deep reinforcement learning with a combinatorial action space for predicting popular reddit threads. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. find
  8. Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. 2016. Model-free episodic control. arXiv preprint arXiv:1606.04460. find
  9. Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics.
  10. Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. 2023. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning. find
  11. Ezgi Korkmaz. 2024. A survey analyzing generalization in deep reinforcement learning. arXiv preprint arXiv:2401.02349. — [arXiv:2401.02349](https://arxiv.org/abs/2401.02349) find
  12. Kangkang Chen, Zhongxue Gan, Siyang Leng, and Chun Guan. 2022. Deep reinforcement learning with parametric episodic memory. In 2022 International Joint Conference on Neural Networks. find
  13. B. A. Kuhl and A. D. Wagner. 2009. Strategic Control of Memory, pages 437–444. Elsevier Ltd. find
  14. Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. 2023. Reward design with language models. arXiv preprint arXiv:2303.00001.
  15. Siwei Chen, Anxing Xiao, and David Hsu. 2023. Llm-state: Open world state representation for long-horizon task planning with large language model. arXiv preprint arXiv:2311.17406. — [arXiv:2311.17406](https://arxiv.org/abs/2311.17406) find
  16. Su Young Lee, Sung-Ik Choi, and Sae-Young Chung. 2019. Sample-efficient deep reinforcement learning via episodic backward update. In Advances in Neural Information Processing Systems. find
  17. Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. 2019. Babyai: A platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations.
  18. Hao Li, Xue Yang, Zhaokai Wang, Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Lewei Lu, and Jifeng Dai. 2024. Auto mc-reward: Automated dense reward design with large language models for minecraft. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. find
  19. Longchao Da, Minquan Gao, Hao Mei, and Hua Wei. 2024. Prompt to transfer: Sim-to-real transfer for traffic signal control with prompt learning. In Thirty-Eighth AAAI Conference on Artificial Intelligence. find
  20. Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, and 1 others. 2022. find
  21. Pre-trained language models for interactive decision-making. In Advances in Neural Information Processing Systems.
  22. Michelle M. Ramey, John M. Henderson, and Andrew P. Yonelinas. 2022. Episodic memory processes modulate how schema knowledge is used in spatial memory decisions. Cognition, 225:105111. find
  23. Zhuo Li, Derui Zhu, Yujing Hu, Xiaofei Xie, Lei Ma, Yan Zheng, Yan Song, Yingfeng Chen, and Jianjun Zhao. 2023. Neural episodic control with state abstraction. In International Conference on Learning Representations. find
  24. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
  25. Dayang Liang, Yaru Zhang, and Yunlong Liu. 2024. Episodic reinforcement learning with expanded state-reward space. In Proceedings of the 23rd International Conference on Autonomous Agents and Multi-agent Systems. find
  26. Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, and Aaron C. Courville. 2021. Pretraining representations for data-efficient reinforcement learning. In Advances in Neural Information Processing Systems. find
  27. Zichuan Lin, Tianqi Zhao, Guangwen Yang, and Lintao Zhang. 2018. Episodic memory deep q-networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. find
  28. Daniel J Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern, and 1 others. 2023. Faster sorting algorithms discovered using deep reinforcement learning. Nature, 618(7964):257–263. find
  29. Hyungho Na, Yunkyeong Seo, and Il-chul Moon. 2024. Efficient episodic memory utilization of cooperative multi-agent reinforcement learning. In International Conference on Learning Representations. find
  30. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.… — Training language models to follow instructions with human feedback
  31. Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, and Chitta Baral. 2024. Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language P…
  32. Shaohui Peng, Xing Hu, Qi Yi, Rui Zhang, Jiaming Guo, Di Huang, Zikang Tian, Ruizhi Chen, Zidong Du, Qi Guo, and 1 others. 2024. Hypothesis, verification, and induction: grounding large language models with self-driven skill learning. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  33. Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. 2017. Neural episodic control. In International conference on machine learning. find
  34. Chak Lam Shek, Xiyang Wu, Wesley A. Suttle, Carl E. Busart, Erin G. Zaroukian, Dinesh Manocha, Pratap Tokekar, and Amrit Singh Bedi. 2024. LANCAR: leveraging language for context-aware robot locomotion in unstructured environments. In IEEE/RSJ International Conference on Intelligent Robots and Syste… find
  35. Ruizhe Shi, Yuyao Liu, Yanjie Ze, Simon Shaolei Du, and Huazhe Xu. 2024. Unleashing the power of pre-trained language models for offline reinforcement learning. In International Conference on Learning Representations. find
  36. Yash Shukla, Wenchang Gao, Vasanth Sarathy, Alvaro Velasquez, Robert Wright, and Jivko Sinapov. 2024. Lgts: Dynamic task sampling using llm-generated sub-goals for reinforcement learning agents. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems. find
  37. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, and 1 others. 2016. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489. find
  38. Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. 2024. Twosome: An efficient online framework to align llms with embodied environments via reinforcement learning. International Journal of Artificial Intelligence and Robotics Research, 01(02):2450004. find
  39. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  40. Luong Quoc Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. 2024. Reft: Reasoning with reinforced fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.
  41. Kathryn Tunyasuvunakool, Jonas Adler, Zachary Wu, Tim Green, Michal Zielinski, Augustin Žídek, Alex Bridgland, Andrew Cowie, Clemens Meyer, Agata Laydon, and 1 others. 2021. Highly accurate protein structure prediction for the human proteome. Nature, 596(7873):590–596. find
  42. Boyuan Wang, Yun Qu, Yuhang Jiang, Jianzhun Shao, Chang Liu, Wenming Yang, and Xiangyang Ji. 2024. Llm-empowered state representation for reinforcement learning. In International Conference on Machine Learning. find
  43. Ruoyao Wang, Peter A. Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. find
  44. Yuqing Wang and Yun Zhao. 2023. Gemini in reasoning: Unveiling commonsense in multimodal large language models. arXiv preprint arXiv:2312.17661. — [arXiv:2312.17661](https://arxiv.org/abs/2312.17661) find
  45. Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom Mitchell. 2023. Read and reap the rewards: learning to play atari with the help of instruction manuals. In Advances in Neural Information Processing Systems. find
  46. Gui Xue. 2018. The neural representations underlying human episodic memory. Trends in Cognitive Sciences, 22(6):544–561. find
  47. Xue Yan, Yan Song, Xinyu Cui, Filippos Christianos, Haifeng Zhang, David Henry Mguni, and Jun Wang. 2023. Ask more, know better: Reinforce-learned prompt questions for decision making with large language models. arXiv preprint arXiv:2310.18127. — [arXiv:2310.18127](https://arxiv.org/abs/2310.18127) find
  48. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Advances in neural information processing systems. find
  49. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations.
  50. Guangxiang Zhu, Zichuan Lin, Guangwen Yang, and Chongjie Zhang. 2020. Episodic reinforcement learning with associative memory. In International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-agentic-episodic-control/